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WHAT  DO  TRAINERS  NEED  TO  KNOW  TO  TRAIN  HIGHER-ORDER  THINKING 
SKIEES? 

EXECUTIVE  SUMMARY 


Research  Requirement: 

This  research  sought  to  characterize  the  complementary  training  needed  by  Army 
instructors  to  support  their  ability  to  train  Soldiers  in  making  decisions  under  uncertainty,  an 
activity  requiring  Soldiers  to  apply  complex,  higher-order  thinking  skills.  Here,  the  researchers 
sought  to  define  what  instructors/trainers  need  to  know  in  order  to  train  Soldiers  in  higher-order 
thinking  skills  using  a  training  needs  analysis  methodology.  Although  the  focus  of  the  training 
needs  analysis  was  on  decision-making  and  problem-solving  training  for  Air  Defense  Artillery 
instructors/trainers,  the  findings  of  the  research  are  broadly  applicable  across  the  Army. 

Procedure: 

Military  and  scientific  literature  was  reviewed  to  identify  instructor  training  relevant  to 
developing  higher-order  thinking  skills  for  decision-making  and  problem-solving.  Observations 
and  interview  data  were  collected  with  Air  Defense  Artillery  (ADA)  subject  matter  experts  at 
Eorts  Sill  (OK),  Hood  (TX),  and  Bliss  (TX)  in  support  of  a  training  needs  analysis.  A  training 
needs  analysis  method  was  used  to  identify  specific  topic  domains  to  include  in  complementary 
training  for  instructors  to  assist  them  in  developing  learners’  higher-order  thinking  skills.  The 
particular  focus  of  the  recommendations  is  on  instructors  at  the  unit  level,  as  these  trainers  are 
most  responsible  for  training  the  complex  tactical  engagements  skills  that  require  Soldiers  to 
apply  higher-order  thinking  to  decision-making  and  problem-solving  scenarios. 

bindings: 

Due  to  the  fact  that  Air  Defense  training  is  highly  technical,  it  often  relies  on  training 
techniques  that  emphasize  remembering,  understanding,  and  applying  technical  systems 
knowledge  and  procedures.  That  said,  in  practice,  ADA  Soldiers  are  required  to  make  sense  of  a 
vast  array  of,  at  times,  incomplete  data  to  visualize  the  battlespace  and  make  complex 
engagement  decisions,  while  weighing  various  contextual  factors  and  adhering  to  the  rules  of 
engagement.  This  requires  ADA  Soldiers  to  engage  in  higher-order  thinking  to  support  their 
decision-making  and  problem-solving  processes.  In  this  report,  we  identify  critical  situations  in 
ADA  that  elicit  complex  decision-making  and  problem-solving  as  well  as  make  more  general 
recommendations  for  what  Army  instructors  need  to  know  in  order  to  train  these  skills.  Our 
recommendations  are  drawn  from  the  scientific  literature  on  training  higher-order  thinking  skills 
and  applying  adult  learning  techniques,  supporting  the  Army’s  current  learning  model. 

Utilization  and  Dissemination  of  bindings: 

As  the  first  in  a  series  of  two  reports,  this  report  covers  the  background  research  supporting  the 
development  of  an  8 -hour  training  program  for  ADA  instructors/trainers  focused  on  how  to  train 
higher-order  thinking  skills.  The  second  report  will  present  the  research  product:  the  8-hr  block 
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of  training.  This  training  was  provided  to  instructors  at  the  ADA  schoolhouse  and  to  unit 
trainers  at  the  108*  ADA  BDE  at  Fort  Bragg,  NC.  The  results  of  the  research  were  briefed  to  the 
U.S.  Army  Air  Defense  Artillery  School  (USAADASCH)  leadership  at  Fort  Sill,  OK,  in  Oct 
2016.  Aspects  of  this  research  were  also  presented  at  the  Department  of  Defense  Fab  Day  Event 
at  the  Pentagon,  Washington,  DC,  in  May  2017. 


V 


WHAT  DO  TRAINERS  NEED  TO  KNOW  TO  TRAIN  HIGHER-ORDER  THINKING 
SKIEES? 

CONTENTS 


Page 

INTRODUCTION . I 

Background  on  the  Research  Problem . 1 

What  Are  Higher- Order  Thinking  Skills? . 2 

Training  Higher- Order  Thinking  Skills . 4 

Higher-Order  Thinking  Skills:  Decision  Making  Under  Uncertainty . 5 

TRAINING  NEEDS  ANAEYSIS  METHOD . 7 

Developing  a  Higher-Order  Thinking  Skills  Training  Needs  Analysis . 8 

TRAINING  NEEDS  ANAEYSIS  RESUETS  AND  DISCUSSION . 11 

Identification  of  Specific  Higher-Order  Thinking  Skills . 12 

Application  of  Training  Needs  Analysis  to  ADA  Crew  Trainers . 14 

Step  1:  Desired  Trainer  Performance . 15 

Step  2:  Current  Trainer  Performance . 16 

Step  3:  The  Gap  in  Trainer  Performance . 17 

Step  4:  The  Causes  of  the  Gap . 18 

Step  5:  Determine  the  Training  Needed  to  Correct  the  Gap . 19 

Step  6:  Potential  Training  Solutions . 19 

TECHNIQUES  TO  ENHANCE  HIGHER-ORDER  THINKING  SKIEES  TRAINING . 20 

Instructional  Approaches  to  Train  Higher-Order  Thinking  Skills . 20 

Instructional  Strategies . 21 

Instructional  Methods . 22 

Instructional  Techniques . 24 

Example  Instructional  Approaches . 26 

CONCEUSIONS . 30 

General  Higher-Order  Thinking  Skills  Training  Recommendations . 30 

ADA  Patriot  Missile  Crew  Training . 30 

REEERENCES . 32 

EIST  OP  ACRONYMS . 38 

APPENDIX  A.  Decision-Making  Diagrams  . A-1 

APPENDIX  B.  Methods  for  Postering  Higher-Order  Thinking  Skills . B-1 


vi 


CONTENTS  (continued) 


Page 

LIST  OF  TABLES 

TABLE  1.  CONTEMPORARY  TERMINOLOGY  DESCRIBING  THINKING  SKILLS . 4 

TABLE  2.  SOURCES  OF  DATA  FOR  THE  TRAINING  NEEDS  ANALYSIS . 9 

TABLE  3.  SPECIFIC  HIGHER-ORDER  THINKING  SKILLS . 13 

TABLE  4.  HIGHER-ORDER  THINKING  SKILLS  EXERCISED  IN  CREW  TASKS . 16 

TABLE  5.  CHARACTERISTICS  OF  THE  REVISED  TRAINING  APPROACH  TO  FOSTER 
HIGHER-ORDER  THINKING  SKILLS  DURING  TRAINING 
IN  THE  MDMP . 27 

TABLE  6.  POTENTIAL  WAYS  TO  FOSTER  HIGHER-ORDER  THINKING  SKILLS 

DURING  TRAINING . 28 

LIST  OF  FIGURES 

FIGURE  1.  THE  DECISION  MAKING  MACHINE . 6 

FIGURE  2.  FACTORS  AFFECTING  DECISION-MAKING . 7 

FIGURE  3.  HIGHER-ORDER  THINKING  SKILLS  TRAINING  NEEDS  ANALYSIS 

FLOWCHART . 10 

FIGURE  4.  FACTOR  CONTRIBUTING  TO  A  PERFORMANCE  GAP . 1 1 


vii 


What  Do  Trainers  Need  to  Know  to  Train  Higher-Order  Thinking  Skills? 

Introduction 


The  Army’s  learning  model  and  human  dimension  initiatives  seek  to  develop  and  refine 
Soldiers’  thinking  skills  in  support  of  problem-solving,  decision-making,  perceptual  processing, 
and  information  processing  (U.S.  Department  of  the  Army,  TRADOC,  2011;  2014).  Current 
Army  training  presents  many  opportunities  to  train  complex  cognitive  skills.  However,  most 
Army  instructors  have  not  been  prepared  to  train  these  skills  (National  Research  Council,  2015). 
This  research  sought  to  identify  what  instructors  need  to  know  in  order  to  develop  trainees’ 
higher-order  thinking  skills.  In  brief,  instructors  should  be  able  to  recognize  when  opportunities 
arise  to  train  these  skills  as  well  as  be  proficient  in  techniques  to  do  so.  By  harnessing  their  own 
and  others’  military  experience,  instructors  should  be  able  to  vary  training  activities  and  problem 
scenarios  realistically.  Trainees  will  be  less  likely  to  become  complacent  in  training  when 
instructors  expose  them  to  varying  and  unexpected  training  activities  and  exercises  (Bransford  & 
Schwartz,  1999). 

Air  Defense  Artillery  instructors/trainers  were  the  initial  target  audience  for  the  products 
of  this  research.  Even  so,  many  of  the  recommendations  we  derived  from  a  training  needs 
analysis  are  broadly  applicable  to  instructors/trainers  who  conduct  training  in  other  areas  of  the 
Army. 

Background  on  the  Research  Prohlem 

The  Army  is  limited  in  terms  of  training  time  and  resources  available.  Training  to 
enhance  complex,  higher-order  thinking  skills  takes  time.  Given  limited  time,  one  could  argue 
that  the  Army  should  focus  on  training  through  repetitive  practice.  Repetitive  practice  enables 
Soldiers  to  learn  to  execute  critical  tasks  quickly,  consistently,  and  proficiently.  Repetitive 
practice  makes  skilled  performance  consistent  for  Soldiers  and  predictable  for  Leaders.  Leaders 
need  to  know  that  Soldiers  can  execute  tasks  proficiently,  even  in  highly  stressful  combat 
conditions. 

A  problem  arises,  however,  when  Soldiers  encounter  novel  situations.  Through  repetitive 
practice,  a  Soldier  gets  good  at  executing  a  specific  skill  in  a  specific  context.  Real-world 
situations  often  do  not  exactly  replicate  those  encountered  in  training.  Soldiers  must  be  able  to 
transfer  what  they  have  learned  into  unforeseen  situations.  This  requires  thinking  skills  that  are 
more  complex  than  those  developed  through  repetitive  practice  alone.  While  training  in  a  way 
that  develops  higher-order  thinking  skills  may  take  more  time,  it  may  have  the  advantage  of 
helping  Soldiers  to  develop  decision-making  and  problem-solving  abilities  that  are  widely 
applicable. 

Creating  the  conditions  to  train  higher-order  thinking  skills  is  directly  associated  with 
applications  of  particular  training  techniques.  Most  instructors/trainers  default  to  using  the 
training  techniques  their  own  trainers  used.  They  know  they  can  apply  these  familiar  techniques 
competently.  Instructors  need  to  be  aware  of  how  higher-order  thinking  skills  can  be  elicited  in 
training  in  order  to  target  them  effectively  (Miri,  David,  &  Uri,  2007).  By  varying  what  Soldiers 
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experience  in  training,  instructors  may  enhance  Soldiers’  ability  to  generalize  principles  from 
what  they  are  learning  (Bransford  &  Schwartz,  1999).  Instructors  must  also  understand  how  to 
communicate  in  ways  that  elicit  complex  cognition  in  the  classroom.  Instructors  need  to  be  able 
to  recognize  and  utilize  the  opportunities  they  encounter  to  create  variation  in  training,  thereby 
disrupting  Soldiers’  expectations  and  engaging  them  in  complex  forms  of  thinking  (Pohl,  2000). 

What  Are  Higher-Order  Thinking  Skills? 

A  traditional  way  to  classify  thinking  skills  is  hierarchically.  Within  a  hierarchical 
scheme,  higher-order  thinking  skills  depend  on  those  of  a  lower-order  (Bloom,  Englehart,  Furst, 
Hill,  &  Krathwohl,  1956).  Lower-order  thinking  skills  include  those  related  to  knowing, 
comprehending,  and  applying  information.  Higher-order  thinking  skills  include  analyzing, 
synthesizing,  and  evaluating  information.  In  other  words,  understanding  a  concept  is  associated 
with  a  lower-order  of  thinking  skills  than  is  combining  a  concept  with  other  concepts  to  solve  a 
real-world  problem.  As  Bloom  et  al.  (1956)  observe,  one  cannot  apply  a  concept  until  one 
knows  and  comprehends  it;  likewise,  one  cannot  analyze  a  concept  until  one  knows, 
comprehends,  and  applies  it. 

The  taxonomy  of  cognition  proposed  in  Bloom  et  al.  (1956)  has  continued  to  evolve  in 
contemporary  psychological  research  (see  Anderson  &  Krathwohl,  2001;  Wilson,  2013).  The 
major  change  from  Bloom’s  original  taxonomy  is  that  researchers  now  refer  to  thinking  skills  as 
verbs  rather  than  nouns:  knowledge  became  remembering',  comprehension  became 
understanding',  application  became  applying',  analysis  became  analyzing',  synthesis  became 
creating',  evaluation  became  evaluating  (Anderson  &  Krathwohl,  2001). 

This  may  at  first  seem  to  be  an  irrelevant  distinction.  Using  verbs  rather  than  nouns 
emphasizes  that  thinking  is  not  an  abstract,  conceptual  thing  that  exists  apart  from  us.  It  is 
instead  what  we  do  constantly  every  day.  As  an  example  of  this  point,  consider  the  meaning  of 
knowledge  in  relation  to  the  meaning  of  remembering.  Knowledge  is  an  abstract  thing  that, 
theoretically,  has  an  amount  that  can  be  measured.  Knowledge  is  an  abstract  concept.  To  know 
something,  however,  is  different.  That  we  know  something  can  be  reflected  in  our  ability  to 
remember  it.  Remembering  is  a  familiar,  everyday  thing  we  do.  As  an  action,  remembering  can 
be  practiced;  through  practice,  one  can  get  better  at  it.  It  is  by  observing  a  person  remembering 
that  we  make  inferences  about  specific  things  they  know,  and  over  time,  their  base  of  knowledge. 
By  naming  thinking  skills  with  verbs,  we  are  able  to  identify  more  clearly  how  we  might  go 
about  improving  individuals’  specific  thinking  skills.  It  is  a  subtle  distinction,  but  an  important 
one  for  contemporary  psychology. 

Some  psychologists,  such  as  de  Bono  (1985),  have  developed  specific  techniques  to 
enhance  thinking  skills.  Many  of  these  techniques  have  been  made  popular  by  the  self- 
improvement  and  secondary  education  literature.  For  instance,  there  is  a  movement  in  American 
secondary  education  to  develop  higher-order  thinking  skills  (Teach  for  America,  201 1).  A 
number  of  classroom  activities  have  been  devised  specifically  to  develop  higher-order  thinking 
skills.  Increasingly  more  complex  skills  such  as  judgment,  critical  thinking,  and  problem  solving 
are  purportedly  improved  by  having  students  engage  in  activities  such  as: 
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•  visualizing  a  problem  by  diagramming  it, 

•  separating  relevant  from  irrelevant  information, 

•  seeking  reasons  and  causes, 

•  justifying  solutions  to  problems, 

•  seeing  more  than  one  side  of  a  problem, 

•  weighing  sources  of  information  based  on  their  credibility, 

•  revealing  assumptions  in  reasoning,  or 

•  identifying  bias  or  logical  inconsistencies. 

In  the  research  literature,  cognitive  processes  have  been  classified  into  increasingly 
complex  terminologies.  For  instance,  Anderson  and  Krathwohl  (2001)  have  distinguished  six 
levels  and  19  categories  of  cognitive  processes.  Table  1  provides  a  synopsis  of  terms  as  well  as 
associated  thinking  skills. 
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Table  1 


Contemporary  Terminology  Describing  Thinking  Skills 

Thinking 

Skill 

Descriptive  Terms 

Additional  Examples 

Create 

Generate  -  hypothesize 

Plan  -  design 

Produce  -  construct 

invent,  compose,  predict,  imagine, 
propose,  devise,  formulate,  combine, 
originate,  forecast,  invent,  assemble 

Evaluate 

Check  -  coordinate,  detect,  test 

Critique  -  judge 

select,  choose,  decide,  justify,  debate, 
verify,  argue,  recommend,  assess, 
discuss,  rate,  prioritize,  determine, 
criticize,  weigh,  value,  estimate,  defend, 
convince,  support,  score 

Analyze 

Differentiate  -  discriminate,  distinguish, 
focus,  select 

Organize  -  find  coherence,  integrate, 
outline,  parse,  structure 

Attribute  -  deconstruct 

examine,  investigate,  separate,  take 
apart,  differentiate,  subdivide,  deduce, 
compare,  contrast,  infer,  calculate, 
monitor 

Apply 

Execute  -  carry  out 

Implement  -  use 

solve,  construct,  compute,  complete, 
make,  put  together,  change,  produce, 
calculate,  manipulate,  modify, 
demonstrate 

Understand 

Interpret  -  clarify,  paraphrase,  represent, 
translate 

Exemplify-  illustrate,  instantiate 

Classify  -  categorize,  subsume 

Summarize  -  abstract,  generalize 

Infer  -  conclude,  extrapolate,  interpolate, 
predict 

Compare  -  contrast,  map,  match 

Explain  -  construct  models 

outline,  discuss,  distinguish,  predict, 
restate,  describe,  relate,  summarize, 
convert,  visualize,  describe,  sketch 

Remember 

Recognize  -  identify 

Recall  -  retrieve 

tell,  list,  locate,  write,  find,  state,  name, 
label,  define,  reproduce,  memorize, 
select,  recite 

Training  Higher- Order  Thinking  Skills 

Higher-order  thinking  skills  enable  us  to  develop  new  knowledge.  That  is,  higher-order 
thinking  skills  enable  learners  to  recognize  and  solve  novel  problems,  as  well  as  adapt  what  they 
have  learned  to  changing  circumstances.  When  faced  with  unfamiliar  situations,  learners  need  to 
be  able  to  apply  skills  related  to  analyzing,  synthesizing,  and  evaluating  information  to  go 
beyond  what  they  currently  know.  It  is  by  way  of  these  skills  that  learners  may  engage  with 
more  complex  knowledge  domains  and  learning  situations  (Paul  &  Nosich,  1992). 
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Enhancing  higher-order  thinking  skills  is  a  common  educational  goal  (Teach  for 
America,  2011).  However,  secondary  education  has  historically  relied  on  techniques  that  elicit 
and  develop  lower-order  thinking  skills:  remembering,  understanding,  and  applying.  A  lot  of 
Army  training  can  be  categorized  similarly.  There  is  a  functional  purpose  to  this  emphasis. 
Teaching  that  is  oriented  to  recalling  and  applying  facts  allows  for  standardized  teaching  and 
evaluation  processes  (Barnes,  2007).  On  a  standard  recall  test,  a  student  can  get  an  answer  right 
(i.e.,  recalling  it  correctly)  or  wrong  (i.e.,  not  recalling  it  correctly).  Performance  outcomes  are 
less  contestable  and  distinctions  among  levels  of  performance  are  clear-cut.  Students  are 
rewarded  for  remembering,  understanding,  and  correctly  applying  facts.  For  younger  learners, 
this  approach  may  be  appropriate;  for  adults,  it  can  be  limiting  (Nihill  &  Stallings,  2014). 

Thinking  is  always  about  something  (James,  1890[1983]).  Teaching  higher-order 
thinking  skills  requires  a  vehicle,  that  is,  content  about  which  the  learner  can  think.  The  content 
needs  to  be  presented  in  a  way  that  encourages  thinking.  To  develop  thinking  skills,  professional 
training  often  makes  use  of  ill- structured  problems  or  case  studies.  These  types  of  training 
materials  present  students  with  situations  that  “require  the  ability  to  recall  useful  information 
quickly,  use  pattern  recognition,  discern  pertinent  information,  think  ahead,  and  anticipate 
outcomes  and  problems  while  remaining  composed  so  that  emotions  do  not  hinder  decision 
making  skills”  (American  Dental  Education  Association,  2015).  Training  higher-order  thinking 
skills  requires  content  and  context  about  which  students  can  think. 

Finally,  many  techniques  for  training  higher-order  thinking  skills  elicit  a  shift  in 
perspective.  For  instructors/trainers  and  learners,  the  shift  calls  attention  to  one’s  own  and 
others’  thinking  processes,  a  phenomenon  called  metacognition  (Flavell,  1979).  Developing 
skills  in  metacognition  may  enable  learners  to  self-regulate,  streamline,  and  ultimately  improve 
their  thinking.  Exposing  learners  to  ill- structured  problems  may  elicit  higher-order  thinking; 
working  with  others  through  contrasting,  sometimes  conflicting,  examples  and  interpretations 
may  extend  this  process  further  (Schwartz,  Chase,  Oppezzo,  &  Chin,  2011). 

Higher-Order  Thinking  Skills:  Decision  Making  Under  Uncertainty 

Organizations  seek  to  improve  thinking  skills  because  they  are  foundational  to  the 
decision-making  process.  A  decision  removes  options  and  locks  us  into  a  set  of  actions  and 
consequences.  We  often  do  not  know  everything  we  need  to  make  a  perfect  decision;  we  decide 
under  conditions  of  uncertainty.  Well-executed  decision-making  mitigates  the  inherent  risk  of 
uncertainty. 

There  are  two  basic  perspectives  on  decision-making:  normative  and  descriptive 
(Kahneman,  2011).  A  normative  approach  to  decision-making  focuses  on  how  people  ought  to 
make  decisions.  It  assumes  that  human  beings  are  ultimately  rational,  motivated  in  our  decisions 
to  get  more  of  what  we  like  and  less  of  what  we  do  not  like.  When  we  deviate  from  a  rational 
basis,  it  is  because  the  world  is  imperfect — we  lack  information,  viable  options,  etc.  Otherwise, 
we  would  always  act  in  predictable  ways  to  obtain  what  we  value.  Descriptive  decision-making 
is  based  on  what  researchers  actually  observe  about  the  behaviors  of  decision-makers  in 
everyday  life.  What  we  actually  do  when  making  a  decision  is  more  important  to  descriptive 
researchers  than  what  we  should  do  according  to  an  abstract  economic  model  (Klein,  2008). 
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When  making  everyday  life  decisions,  it  would  seem  that  we  decide  based  on  what  we 
perceive  in  a  situation,  and  how  we  relate  what  we  perceive  to  our  experiences  (Klein,  2008). 

We  often  make  quick,  error-prone,  even  risky  decisions.  To  counteract  this  risk,  we  may  offload 
aspects  of  the  evaluative  process  using  techniques — like  the  Military  Decision  Making  Process — 
that  tend  to  frame  a  decision  in  a  systematic,  almost  mathematical,  way.  Being  thorough  and 
precise  takes  time.  For  some  decisions,  however,  there  is  not  enough  time.  Decision-making 
needs  to  be  quick  and  intuitive  in  some  critical  situations,  methodical  and  precise  in  others. 
Ideally,  a  decision-maker  will  develop  skill  in  both  approaches  to  decision-making  and  know 
when  to  apply  the  respective  techniques  appropriately,  engaging  in  self-regulation  around  the 
decision-making  process.  Figure  1  presents  a  general  decision-making  process. 


The  Decision  Making  Machine 


Identification 

Unit 


Environment 

Estoblishment 

(Relationships 
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Figure  1 .  The  Decision  Making  Machine 

Decisions  are  guided  by  internal  and  external  factors  about  which  decision-makers  may 
have  inconsistent  information  (Luft,  1970).  When  we  realize  our  information  is  incomplete  or  of 
uneven  quality,  time  is  limited,  the  situation  changes  during  the  decision  process,  and/or  there  is 
too  much  irrelevant  information  (i.e.,  noise),  decision-making  can  become  stressful  (Zhang,  Ji,  & 
Looney,  2002).  One  way  to  help  decision-makers  become  less  reactive  to  this  stress  is  to  expose 
them  in  training  to  progressively  more  uncertain  conditions  with  increasingly  higher  stakes, 
followed  by  an  after-action  review  (AAR).  This  technique  provides  a  stress-free  context  in 
which  to  reflect  on  one’s  decision-making  process  while  being  stressed. 

Internal  and  external  factors  that  influence  decision-making  are  apparent  in  varying 
degrees.  External  factors  that  are  immediately  present  in  our  environment  and  that  are  directly 
influencing  our  task  tend  to  be  those  that  can  readily  be  perceived.  Internal  factors  are  less 
apparent.  Nonetheless,  internal  factors  may  influence  the  decision-making  processes.  Internal 
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factors  include  existing  experiences  and  skills,  personality  traits,  and  preferred  ways  of 
approaching  problems  (i.e.,  heuristics)  and  preferred  ways  of  interpreting  things  (i.e.,  biases).  It 
is  often  very  difficult  to  manipulate  internal  factors  directly.  So,  effective  trainers  must 
manipulate  external  factors  in  the  world  (e.g.,  characteristics  of  the  problem-set  or  context)  in 
order  to  elicit  and  develop  the  internal  factors  guiding  the  decision-maker.  Figure  2  depicts  the 
process  how  decision-makers  collect  information  and  the  influences  of  internal  and  external 
factors  on  the  decision-making  process. 


Figure  2.  Factors  Affecting  Decision-Making.  For  additional  detail  on  this  figure,  see  Appendix 
A. 


Training  Needs  Analysis  Method 

To  address  the  question  of  what  instructors/trainers  need  to  know  in  order  to  train 
Soldiers’  higher-order  thinking  skills,  we  conducted  a  training  needs  analysis  (TNA).  This  TNA 
focused  on  the  training  provided  to  ADA  Patriot  crews.  Like  Soldiers  in  other  MOSs,  ADA 
crews  face  ambiguous  situations  in  which  they  must  keep  aware  of  what  is  happening  and  make 
decisions  based  on  evolving  information.  Patriot  crews  evaluate  multiple  aerial  contacts, 
determine  whether  they  are  friendly  or  not,  and  assign  air  defense  assets  to  those  determined  to 
be  unfriendly.  The  information  used  to  determine  what  to  do  comes  from  a  variety  of  sources — 
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e.g.,  radar  arrays,  communications  with  other  personnel.  An  initial  plan  undergoes  continuous 
revision,  given  new  information  and  changing  priorities. 

ADA  crews  may  benefit  from  training  in  higher-order  thinking  skills.  Improving  these 
skills  may  support  them  in  interpreting  ambiguous  information  and  making  effective  decisions  in 
response.  Associated  with  that  need  is  one  specific  to  trainers.  Trainers  need  to  know  how  to 
train  higher-order  thinking  skills.  Currently,  trainers  receive  no  instruction  on  effective  ways  to 
elicit  and  develop  Soldiers’  decision-making  skills  for  ambiguous  situations.  Since  ADA 
Schoolhouse  training  is  primarily  focused  on  initializing  the  Engagement  Control  Station 
(ECS)/Information  Coordination  Central  (ICC),  air  battle  operations — where  most  critical 
decision-making  occurs — becomes  the  responsibility  of  unit  trainers  at  the  Patriot  battalion  and 
battery.  While  Schoolhouse  instructors  receive  instructor  skills  training  and  teach  from  a 
standardized  curriculum,  unit  trainers  often  must  devise  their  own  approach  to  training  and 
certifying  their  crews. 

There  appears  to  be  a  gap  in  the  preparation  unit  trainers  receive,  particularly  with  respect 
to  training  decision-making  and  problem-solving  skills  to  be  applied  in  ambiguous  contexts.  To 
address  this  gap,  a  TNA  was  conducted.  TNA  is  a  technique  that  contrasts  desired  performance 
with  current  performance  for  individuals  and  organizations  (Miller  &  Osinski,  1996).  TNA 
helps  determine  what  training  is  needed  to  bridge  identified  gaps  (Goldstein,  1993).  Sometimes 
gaps  can  be  addressed  with  training,  other  times  gaps  are  due  to  personnel  or  material  resources 
(U.S.  Office  of  Personnel  Management,  2013).  A  TNA  begins  by  matching  performance 
requirements  to  specific  knowledge,  skills,  and  abilities.  Then,  if  those  knowledge,  skills,  and 
abilities  are  determined  not  to  be  present,  training  can  be  developed  to  address  the  gap. 

Developing  a  Higher-Order  Thinking  Skills  Training  Needs  Analysis 

A  review  of  literature  on  higher-order  thinking  skills  assisted  in  identifying  the 
knowledge,  skills,  and  abilities  required  by  trainers  to  develop  trainees’  higher-order  thinking 
skills.  A  list  of  key  skills  was  assembled.  Eor  the  TNA,  we  targeted  those  skills  associated  with 
the  cognitive  tasks  of  analyzing,  evaluating,  and  creating,  i.e.,  higher-order  thinking  skills. 

Training  Needs  Analysis  for  ADA  Patriot  Missile  Crew  Trainers.  The  TNA 

identified  training  needs  for  unit  trainers  of  Patriot  Missile  crews.  Data  were  collected  from 
reviews  of  ADA  literature,  observation  of  Patriot  missile  crew  pre-certification  training  and 
AARs,  interviews  with  crew  certification  personnel,  interviews  with  ADA  School  training 
personnel,  and  discussions  with  Patriot  missile  crewmembers.  Table  2  lists  the  personnel 
interviewed  and  the  primary  type  of  information  gained  for  use  in  the  TNA. 
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Table  2 


Sources  of  Data  for  the  Training  Needs  Analysis 


Source 

Number  of 
Personnel 

Method  of 
Collection 

Information 

ADA  School  Senior  Warrant 

Officer  (CW5)  -  Ft.  Sill,  OK 

1 

Interview 

Patriot  missile  crew  training, 
trainer  training,  common  training 
issues 

ADA  School  Course  Managers  and 
Course  Directors  -  Ft.  Sill,  OK 

7 

Interview 

Training  content  within  various 
courses 

ADA  School  Instructors  (Off,  WO, 
NCO)  -  Ft.  Sill,  OK 

7 

Interview 

Training  content,  methods,  and 
techniques 

31^'  ADA  BDE  SMEs  (Btry 

Trainer,  EMMO/Standardization 

Off,  NCO)  -  Et.  Sill,  OK 

4 

Interview 

Patriot  missile  crew  training, 
trainer  training,  trainer  status, 
training  methods  and  techniques 

69*  ADA  BDE  SMEs  (Off,  WO, 
NCO,  Enlisted)  (Patriot  missile 
crews  &  trainers)  -  Et.  Hood,  TX 

12  3 -person 
crews 
(36  total) 

Interview, 
observation 
of  AARs  & 
operations 

Patriot  missile  crew  operations. 
Patriot  missile  crew  training,  crew 
training  status 

SPEAR  Team  SMEs  (Off,  CWO, 
NCO,  contractors)  -  Et.  Hood,  TX 

20 

Interview, 
observation 
of  AARs  & 
operations 

Patriot  missile  crew  training,  crew 
training  status,  common  training 
issues 

Note:  Chief  Warrant  Officer  5  (CW5),  Warrant  Officer  (WO),  Officer  (Off),  Noncommissioned  Officer  (NCO), 
Brigade  (BDE),  Battery  (Btry),  Electronic  Missile  Maintenance  Officer  (EMMO),  Standardized  Patriot  Evaluation 
and  Assessment  Reporting  (SPEAR),  Subject  Matter  Expert  (SME),  After  Action  Review  (AAR). 


For  the  TNA,  desired  and  current  trainer  performances  were  examined  through  a 
combination  of  observation  and  discussions  with  trainers  and  senior  ADA  training  personnel.  In 
addition,  these  personnel  helped  to  identify  the  specific  higher-order  thinking  skills  used  in 
Patriot  Crew  Operations,  which  would  benefit  from  additional  training.  In  addition,  researchers 
observed  ADA  Patriot  crews  inside  the  Engagement  Control  Station  as  they  fought  simulated  air 
battles  during  their  pre-certification  training  as  well  as  the  associated  AARs  conducted  by 
training  personnel. 

Various  techniques  can  be  applied  in  conducting  a  TNA  (cf.  U.S.  Office  of  Personnel 
Management,  2013;  Morrison,  2012;  Chand,  2015).  Here,  the  researchers  first  identified  the 
specific  higher-order  thinking  skills  to  be  exercised  during  training.  Once  the  key  thinking  skills 
were  identified,  developmental  concepts  the  trainers  need  to  apply  were  assessed,  the  factors 
contributing  to  the  gaps  were  identified,  the  training  needed  to  address  the  gap  were  determined, 
and  solutions  to  how  that  training  can  be  obtained  were  developed.  Figure  3  presents  the 
approach  utilized  in  this  research  to  identify  what  trainers  need  to  know  to  develop  Soldiers’ 
higher-order  thinking  skills. 
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Figure  3.  Higher-Order  Thinking  Skills  Training  Needs  Analysis  Flowchart.  The  HOTS 
acronym  refers  to  higher-order  thinking  skills. 

Step  1:  Identification  of  Desired  Trainer  Performance.  In  conducting  the  TNA,  the 
researchers  first  identified  higher-order  thinking  skills  concepts  that  trainers  should  be 
implementing  during  training,  including  those  the  trainer  should  target  when  delivering 
instruction.  This  step  included  identifying  opportunities  trainers  may  use  to  assist  learners  in 
developing  their  existing  critical  thinking  and  problem-solving  skills. 

Step  2:  Identify  Current  Trainer  Performance.  After  specific  thinking  skills  were 
identified,  the  researchers  observed  and  interviewed  trainers  in  the  classroom.  Particularly 
important  was  determining  whether  instructors  were  putting  into  practice  adult  learning 
techniques  to  elicit  and  support  development  of  higher-order  thinking  skills. 

Step  3:  Identify  the  Gap  in  Trainer  Performance.  To  identify  the  gap  in  trainer 
performance,  the  researchers  compared  desired  performance  with  current  performance.  Where 
discrepancies  arose,  targets  for  additional  training  were  identified. 

Step  4:  Determine  the  Cause  of  the  Performance  Gap.  Once  gaps  were  identified,  the 
researchers  sought  out  potential  causes.  Factors  often  associated  with  training  gaps  for  higher- 
order  thinking  skills  included: 

•  insufficient  trainer  preparation  for  training  higher-order  thinking  skills, 

•  lack  of  understanding  of  how  to  incorporate  development  concepts, 

•  improper  use  of  available  training  time, 

•  thinking  skills  training  not  designed  as  part  of  the  training  program, 

•  no  value  assigned  to  thinking  skills  due  to  lack  of  understanding,  or 

•  causes  beyond  the  trainers’  influence  (e.g.,  personnel  or  resources). 
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Additional  factors  may  include  student  differences,  student  performance  requirements, 
and  training  environment  factors  (e.g.,  course  content,  resources,  and  the  flexibility  to  tailor 
training)  (Dyer,  Wampler,  &  Blankenbeckler,  2011).  Figure  4  shows  how  factors  may  interact, 
contributing  to  the  performance  gap. 


Figure  4.  Factors  Contributing  to  a  Performance  Gap. 

Step  5:  Determine  the  Training  Needed  to  Correct  Performance  Gaps.  After  causes 
of  the  gap  were  determined  to  be  training-related,  training  needs  were  identified.  Training  needs 
were  targeted  in  terms  of  instructional  competencies  to  support  trainers  in  developing  learning 
higher-order  thinking  skills  (Nihill  &  Stallings,  2014;  see  also  Duron,  Limbach,  &  Waugh, 
2006). 


Step  6:  Identify  Potential  Training  Solutions.  The  last  step  in  the  TNA  identified 
solutions  for  how  to  address  identified  training  needs.  Those  solutions  included  specifying  what 
training  should  be  provided  to  trainers,  how  that  training  will  be  given,  and  where  that  training 
will  take  place. 


Training  Needs  Analysis  Results  and  Discussion 

Given  the  specific  content  being  trained,  it  is  unlikely  trainers  will  need  to  develop  all 
higher-order  thinking  skills  in  learners.  Trainers  who  plan  to  develop  thinking  skills  in  learners 
may  need  a  training  curriculum  that  includes  the  following  three  main  instructional 
competencies: 
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•  Higher-Order  Thinking  Skills  Background  Information.  This  training  would  introduce 
trainers  to  higher-order  thinking  skills  concepts  and  their  general  application  to  the 
content  domain  being  taught.  This  training  would  include  information  on  cognitive 
skills,  critical  thinking,  differences  in  higher-  and  lower-order  thinking  skills,  why 
development  of  thinking  skills  in  learners  is  important,  how  thinking  skills  are  elicited, 
and  how  to  create  a  learning  environment  to  encourage  critical  thinking. 

•  Adult  Learning  Methods  and  Techniques.  This  training  would  familiarize  trainers  with 
adult  learning  theory,  training  delivery  methods,  and  instructional  techniques  to  promote 
critical  thinking  and  problem-solving.  Delivery  methods  could  include  collaborative, 
experiential,  exploratory,  and/or  discovery  learning,  use  of  problem-solving  scenarios, 
and  other  methods  that  promote  the  use  of  critical  thinking  skills.  Instruction  on 
techniques  may  include  facilitation,  leading  discussions,  and  effective  questioning. 

•  Higher-Order  Thinking  Skills  for  Targeted  Instruction.  This  training  would  provide 
information  on  how  the  specific  higher-order  thinking  skills  apply  to  the  specific  training 
content  presented  in  a  course.  The  instruction  would  target  how  critical  thinking  and 
problem-solving  skills  are  applied  operationally. 

Identification  of  Specific  Higher-Order  Thinking  Skills 

Burns  et  al.  (2006)  categorized  thinking  skills  into  seven  basic  categories,  which  are 
intended  to  be  generalizable  across  content  domains.  Based  on  the  nomenclature  presented  in 
Table  1,  a  list  of  specific  higher-order  thinking  skills  was  developed  by  relating  cognitive  skills 
associated  with  analyzing,  evaluating,  and  creating  with  Burns  et  al.’s  model.  Table  3  presents 
the  results  of  this  synthesis,  including  descriptions  of  how  and  why  the  selected  cognitive  skills 
are  exercised  (Bums  et  ah,  2006). 


12 


Table  3 

Specific  Higher-Order  Thinking  Skills 


Specific  Higher-Order  Thinking  Skills 

Analyzing 

Evaluating 

Creating 

Meaning 

Breaking  information  into  parts 
to  explore  understandings, 
meanings  and  relationships 

Justifying  a  decision  or 
course  of  action 

Generating  new  ideas, 
products  or  ways  of 
viewing  information  or 
concepts 

Skills  Exercised 

Learner  breaks  information  into 

Learner  conducts  in-depth 

Learner  uses  what  was 

(How) 

parts  to  understand  pieces 

reflection,  criticism  and 
assessment 

previously  known  or 
learned 

Skills  Exercised 
(Why) 

To  identify  evidence  for  a 
conclusion 

To  make  a  decision 

To  create  new  ideas  and 
information 

Specific  Skills 

1.  Determining  cause  and 

5.  Deciding:  check. 

6.  Planning  and 

and  Associated 

effect:  deduce,  infer,  calculate. 

coordinate,  detect,  test. 

producing:  generate. 

Action  Verbs 

deconstruct,  take  apart 

2.  Comparing  and  contrasting: 
differentiate,  discriminate 

3.  Classifying:  distinguish, 
select,  organize,  find  coherence, 
parse,  attribute,  structure, 
outline,  integrate,  separate, 
differentiate,  subdivide 

4.  Observing  and  examining: 
investigate,  focus,  monitor 

critique,  judge,  select, 
choose,  justify,  debate, 
verify,  argue,  recommend, 
assess,  discuss,  rate, 
prioritize,  determine, 
criticize,  weigh,  value, 
estimate,  defend, 
convince,  support,  score 

design,  create,  invent, 
compose,  construct,  devise, 
invent,  assemble, 
formulate,  combine, 
originate 

7.  Predicting:  hypothesize, 
imagine,  propose,  forecast 

Determining  Cause  and  Effect.  This  cognitive  skill  concerns  the  ability  to  recognize 
linkages  among  events  in  a  sequence.  It  is  also  essential  to  recognizing  that  later  events  were 
contingent  on  those  that  came  before  (Bums  et  al.,  2006). 

Comparing  and  Contrasting.  Comparing  and  contrasting  consists  of  perceiving  the 
relevant  features  of  two  (or  more)  things  and  determining  the  similarities  and  differences 
between  them.  Relevant  features  may  be  physical  (such  as  size,  shape,  color,  or  weight)  or  less 
tangible  (such  as  value,  beauty,  or  plausibility)  (Burns  et  al.,  2006). 

Classifying.  Classifying  involves  determining  the  membership  of  a  particular  object  in  a 
group  based  on  characteristics  it  shares  with  other  objects  already  in  that  group.  Classifying 
allows  the  thinker  to  work  with  a  large  number  of  individual  data  points  by  grouping  the  data  in 
terms  of  defining  characteristics  (Burns  et  al.,  2006). 

Observing  and  Examining.  Observing  and  examining  involve  using  one  or  more  of  the 
five  senses  to  conduct  an  inspection  of  an  item  or  event  to  collect  associated  data.  The  data 
collected  supports  other  higher-order  thinking  skills.  The  data  collected  may  be  qualitative  (e.g. 
seeing  color,  feeling  hot/cold),  or  quantitative  (e.g.,  counting  occurrences  of  an  event  or  tracking 
time)  (Bums  et  al.,  2006).  In  Patriot  operations,  crewmembers  are  expected  to  make  engagement 
decisions  based  on  information  that  comes  to  them  from  multiple  sources,  including  radar  and 
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communications  with  other  command  and  control  elements.  Based  on  this  information,  they 
must  make  sense  of  what  is  happening  in  the  battle  space  and  be  prepared  to  evaluate  and 
reevaluate  their  understanding  of  the  air  battle  based  on  changing  information. 

Deciding.  Deciding  is  a  cognitive  process  in  which  alternative  options  are  perceived, 
evaluated,  and  then  one  option  is  selected.  Typically,  decision-making  is  a  complex  process  that 
culminates  in  taking  some  type  of  action  that  may  limit  future  options.  Most  decisions  are  made 
with  the  information  available  at  the  time,  which  is  often  incomplete  (Bums  et  ah,  2006). 

Planning  and  Producing.  Planning  is  identifying  the  steps  and  actions  to  be  taken  to 
achieve  a  goal.  Producing  is  the  use  of  existing  knowledge  to  create  new  knowledge,  ideas,  or 
concepts.  Inherent  to  both  planning  and  producing  is  the  creation  of  new  knowledge  such  as  a 
new  plan  of  action  or  a  new  understanding  of  an  existing  problem.  Planning  and  producing  are 
both  creative  cognitive  skills,  involving  critical  thinking  in  the  context  of  visualizing, 
hypothesizing,  and  formulating  new  knowledge,  including  novel  solutions  to  problems  or 
improved  concepts  (Bums  et  ah,  2006). 

Predicting.  Predicting  is  the  skill  of  using  past  and  present  available  information  to 
forecast  events  in  the  future.  While  the  outcome  of  future  events  are  never  a  certainty, 
understanding  causal  relationships  and  the  likelihood  that  two  events  will  co-occur  can  help 
predictions  be  more  accurate.  As  predictions  are  based  on  evidence,  they  tend  to  be  more 
valuable  than  guessing.  Predictions  look  for  trends  in  data,  i.e.,  the  patterns  and  tendencies  that 
indicate  a  likely  outcome  (Bums  et  ah,  2006). 

Application  of  Training  Needs  Analysis  to  ADA  Crew  Trainers 

Patriot  missile  crews  must  make  quick  decisions  based  on  incomplete  information  when 
evaluating  multiple  aerial  contacts.  While  the  content  domain  is  primarily  procedural,  critical 
thinking  and  decision-making  skills  are  significant  components.  The  ADA  School  identified  the 
training  environment  as  not  fostering  the  development  of  higher-order  thinking  skills.  The  ADA 
School  is  concerned  that  trainers  may  not  be  specifically  prepared  to  develop  thinking  skills  in 
ADA  crews.  With  proper  instruction,  trainers  may  be  better  prepared  to  address  these  skills 
when  training  ADA  Crews. 

At  the  ADA  School,  crewmembers  are  trained  to  initialize  the  system  and  are 
familiarized  with  Engagement  Control  Station  (ECS)  and  Information  and  Coordination  Central 
(ICC)  van  operations  with  limited  experience  in  crew  operations.  In  practice,  unit  trainers, 
predominately  the  Standardization  Officer  (SO)  and  the  Battery  Trainer  in  the  Patriot  missile 
battery,  are  responsible  for  the  large  majority  of  crew  operator  training.  Although  official 
modified  table  of  organization  and  equipment  (MTOE)  positions  do  not  exist  in  a  Patriot  missile 
battery  for  the  SO  or  the  Battery  trainer,  the  unofficial  positions  are  crucial  to  a  Battery’s 
preparedness  for  executing  Air  Missile  Defense  (AMD)  operations.  The  SO,  normally  a  Warrant 
Officer,  is  the  chief  trainer  in  a  Patriot  missile  battalion.  The  Battery  trainer,  normally  a 
Eieutenant,  holds  the  position  of  the  subject  matter  expert  for  all  matters  related  to  Patriot  missile 
fire  control.  The  SO,  along  with  the  Battery  trainer,  develops  the  operations  and  engagements 
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(tactics)  training  program  for  the  battalion,  maintains  standards,  executes  training  plans,  and 
evaluates  the  proficiency  of  the  ECS/ICC  crews. 

The  six-step  TNA  process  (Figure  3)  was  used  to  examine  potential  trainer  issues 
associated  with  the  ADA  School’s  concern  and  potential  training  solutions. 

Step  1:  Desired  Trainer  Performance 

The  primary  mission  of  unit  trainers  of  ECS  and  ICC  Patriot  missile  crewmembers  is  to 
train  the  actions  required  to  fight  air  battles,  including  Weapons  Control  and  Friendly  Protection. 
Many  tasks  performed  during  these  operations  require  higher-order  thinking  skills.  Moreover, 
Patriot  missile  crewmembers  are  required  to  make  time-sensitive  decisions  using  incomplete 
information  while  dynamically  evaluating  multiple  aerial  contacts.  Dynamic  evaluation  involves 
both  procedural  skills  and  cognitive  skills  to  make  sense  of  incomplete  information,  changing 
information,  multiple  tasks  requiring  attention,  and  time  pressure.  ADA  crews  require  training 
on  both  performing  technical  procedures  and  how  to  use  thinking  skills  to  perform  under 
pressure  with  incomplete  information. 

Interviews  with  Subject  Matter  Experts.  Information  from  SPEAR  evaluators,  senior 
trainers,  and  other  SMEs  was  combined  with  observations  of  crew  operations  to  identify 
situations  in  which  Patriot  crewmembers  would  be  exercising  higher-order  thinking  skills  during 
operations.  In  many  of  the  identified  Patriot  missile  crew  tasks,  the  crew  analyzes  incoming 
information  to  draw  conclusions.  Critical  decisions  are  made  after  some  form  of  analysis; 
likewise,  planning  and  predicting  include  both  analysis  and  decision-making.  In  Table  4,  the 
higher-order  thinking  skill  associated  with  the  ultimate  objective  for  each  of  the  crew  tasks  is 
represented  with  an  “O.”  The  higher-order  thinking  skill  represented  with  an  “X”  is  associated 
with  where  the  predominant  portion  of  thinking  occurs. 
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Table  4 

Higher-Order  Thinking  Skills  Exercised  in  Crew  Tasks. 


Thinking  Skills  Exercised 

O  -  Objective 

X  -  Predominant  Critical  Thinking  Skill 

Analyzing 

Evaluating 

Creating 

Determining  Cause  and 
Effect 

Comparing  and 
Contrasting 

Ciassifying 

Observing  and 
Examining 

Deciding 

Pianning  and  Producing 

Predicting 

Related  Crew  Tasks 

Set  and  revise  task  priorities 

x/o 

Classify  unknown  or  misclassified  track 

X/O 

Recognize  a  developing  self-defense  situation 

X/O 

Identify  a  track  for  auto  engagement  or 
execute  a  manual  engagement 

X 

o 

Request/recommend  modification  of  radar 
search  mode  and/or  area 

X 

X 

o 

Request  assistance  (i.e.  track  ID,  hostile 
classification,  etc.) 

X 

o 

Eliminate/ignore  a  non-hostile  or  non¬ 
threatening  hostile  track 

X 

X 

o 

Plan/alert  battery  or  ICC  to  future  actions 
(reload,  maintenance,  etc.) 

X 

X 

o 

Predict  track  separation  (multiple  ACs, 

ARMS,  ASMs) 

X 

o 

Predict  a  likely  hostile  track  or  action 

X 

o 

Step  2:  Current  Trainer  Performance 

The  researchers  identified  trainer  practices  based  on  discussions  and  interviews  with 
former  SOs  and  Battery  trainers.  In  addition,  observations  of  current  training  were  conducted. 
The  researchers  were  unable  to  witness  actual  training  of  ECS  and  ICC  crews  by  their  primary 
trainers.  Instead,  the  observations  focused  on  crew  performance  during  a  simulated  air  battle 
operation,  a  reflection  of  their  training,  during  crew  certification  conducted  by  SPEAR 
evaluators. 

ADA  crew  training  is  primarily  conducted  at  the  unit  level,  either  in  a  classroom  setting, 
in  a  simulator,  or  in  the  ECS/ICC  van  itself.  Classroom  training  focuses  on  delivery  of 
information  in  which  learners  must  memorize  facts  and  learn  procedures  for  later  application  in 
operations.  The  training  elicits  lower-order  thinking  skills  primarily,  i.e.,  remembering  and 
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understanding.  Once  in  the  simulator  or  van,  crew  training  emphasizes  application  of  knowledge 
and  procedures  while  fighting  scenario-driven  air  battles.  The  trainer  observes  various  crew 
tasks,  makes  on-the-spot  corrections,  and  may  provide  limited  AARs. 

ADA  crews  become  proficient  through  repeated  practice,  using  these  training  techniques 
and  methods.  They  learn  to  interpret  information  on  screens,  manipulate  switches  and  buttons, 
and  apply  procedural  steps.  As  a  result,  crews  become  faster  and  more  accurate  with  procedures. 
Trainers  reported  that  they  routinely  use  this  method  of  training  and  do  not  focus  on  exercising 
higher-order  thinking  skills.  Often,  trainers  reported  that  limited  time  and  crew  turnover  drive 
what  and  how  they  train. 

While  crews  displayed  varying  levels  of  proficiency  in  crew  operations  when  they  were 
exposed  to  scenarios  with  increased  ambiguity  and  complexity,  their  decision-making  abilities 
decreased.  In  this  situation,  many  crews  failed  to  execute  all  necessary  actions.  Planning  and 
predicting  actions  suffered  as  well  when  requirements  increased  and  crew  failed  to  prioritize 
actions  appropriately.  The  SMEs  interviewed  noted  that  even  though  crewmembers  are 
proficient  in  procedural  actions  and  system  manipulation,  they  did  not  always  understand  the 
reason  for  their  actions  or  the  overall  effect  of  those  actions  on  operations. 

Problems  arise  during  operations  when  complexity,  time-pressure,  and  incomplete 
information  are  present.  Under  these  conditions,  ADA  crews  become  less  able  to  identify  a 
proper  course  of  action,  take  longer  to  make  a  decision,  and  sometimes  fail  to  appropriately  plan 
for  future  actions. 

Step  3:  The  Gap  in  Trainer  Performance 

From  the  information  gathered  in  Step  2,  it  was  apparent  that  trainers  routinely  utilize 
training  techniques  that  emphasize  lower-order  thinking  skills:  remembering,  understanding,  and 
then  applying  information.  For  the  majority  of  ADA  crew  training,  development  of  higher-order 
thinking  skills  is  not  an  intended  part  of  the  design.  Trainers  are  not  specifically  using  adult 
learning  methods  to  foster  critical  thinking  and  decision-making  skills  required  by  the 
operational  duties  of  ECS  and  ICC  crews.  Three  issues  were  identified  as  training  gaps: 

•  Higher-Order  Thinking  Skills  Not  Fostered  During  Training.  The  development  of 
higher-order  thinking  skills  in  crewmembers  has  not  been  designed  into  the  Battery 
training  program. 

•  Adult  Learning  Techniques  Not  Used.  Most  training  consists  of  information  and 
procedures  to  be  remembered  and  then  applied.  Repeated  practice  of  procedural 
skills  during  simulated  air  battles  is  the  routine  method  of  training.  Adult  learning 
methods  and  training  techniques  are  not  used  to  foster  higher-order  thinking  skills  in 
learners. 

•  Training  Focuses  on  Application  of  Procedural  Actions  Without  Understanding. 
Some  trainers  focus  on  procedural  actions  without  complete  supporting  information. 
Crews  may  more  effectively  develop  decision-making,  analyzing,  planning,  and 
predicting  skills  when  they  understanding  the  meaning  and  effect  of  procedures  in  the 
context  of  operations. 
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Step  4:  Causes  of  the  Gap 


Based  on  interviews  with  ADA  School  managers  and  senior  personnel,  the  researchers 
explored  why  unit  trainers  use  the  methods  and  techniques  they  currently  employ.  Additional 
information  was  provided  by  schools  who  instruct  Officer  and  Warrant  Officer  personnel  to 
become  SOs  and  Battery  trainers. 

Prior  to  arriving  at  a  unit,  SOs  and  Battery  trainers  receive  no  preparation  to  be  trainers. 
Formal  instruction,  such  as  the  Army  Basic  Instructor  Course  (ABIC),  is  required  only  for 
personnel  destined  to  be  a  trainer  in  school  or  institutional  settings  such  as  at  the  ADA  school. 
Therefore,  unit  trainers  do  not  receive  in-depth  instruction  on  current  Army  Learning  Model 
concepts,  methods,  and  techniques.  The  ADA  School  prepares  Officers  and  Warrant  Officers  for 
their  operational  duties  at  an  ADA  unit,  but  does  not  provide  train-the-trainer  instruction  on  adult 
learning  methods  and  techniques. 

Prior  to  attending  advanced  ADA  schooling,  WOs  attend  training  at  the  Warrant  Officer 
Career  College  (WOCC)  in  Fort  Rucker,  AL.  The  WOCC  trains  all  warrant  officers  through  the 
Warrant  Officer  Candidate  School  (WOCS)  and  the  Warrant  Officer  Intermediate  Level 
Education  (WOILE)  course.  In  a  previous  effort,  Nihill  and  Stallings  (2014)  conducted  research 
concerning  the  use  of  adult  learning  methods  and  techniques  at  the  WOCC.  While  that  effort 
was  designed  to  assist  WOCC  instructors  in  using  adult  learning  concepts  at  the  school, 
discussions  with  the  Director  of  Education  and  Training  provided  that  the  instruction  given  to 
WOs  attending  the  school  did  not  include  train-the-trainer  training  or  information  on  adult 
learning  concepts. 

Personnel  assigned  as  SOs  and  Battery  trainers  are  similar  in  that  the  unit  trainer 
assignment  is  an  additional  duty.  MTOE  authorizations  for  them  do  not  exist.  Often,  unit 
trainers  are  selected  based  on  who  in  the  unit  is  the  most  knowledgeable  and  experienced  in 
employing  the  Patriot  missile  system.  Once  those  duties  are  assigned,  the  unit  trainers  receive  no 
training  to  prepare  them  to  train  crews.  As  such,  they  tend  to  train  as  they  were  trained  and  how 
previous  unit  trainers  trained. 

Discussions  with  current  and  former  SOs  and  Battery  trainers  substantiated  that  they  were 
often  not  familiar  with  higher-order  thinking  skills  concepts  and  more  specifically  decision¬ 
making  as  a  thinking  skill  concept.  They  tended  to  be  unaware  of  the  need  to  establish  higher- 
order  thinking  skills  as  a  training  objective.  As  unit  trainers  were  not  aware  of  the  need,  nor 
focused  on  improving  decision-making  during  training,  they  did  not  have  a  clear  understanding 
of  how  to  develop  higher-order  thinking  skills  in  learners.  This  gap  was  attributed  to  five  causes: 

•  Eack  of  knowledge  on  higher-order  thinking  skills, 

•  Eack  of  knowledge  on  decision-making,  planning  and  predicting, 

•  Eack  of  awareness  of  the  need  for  higher-order  thinking  skills  training, 

•  Eack  of  training  on  adult  learning  methods  and  techniques,  and 

•  Eack  of  training  on  how  to  foster  higher-order  thinking  skills  during  training. 
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Step  5:  Determine  the  Training  Needed  to  Correct  the  Gap 

To  close  the  identified  training  gap,  trainers  will  need  both  informational  and  application 
training  regarding  higher-order  thinking  skills,  adult  learning,  and  the  specific  thinking  skills 
related  to  crew  operations.  Since  virtually  no  training  has  been  provided  to  unit  trainers  to 
prepare  them  to  develop  higher-order  thinking  in  learners,  they  require  a  curriculum  including  all 
three  main  instructional  competencies:  higher-order  thinking  skills  background  information, 
adult  learning  methods  and  techniques,  and  targeted  information  on  the  specific  higher-order 
thinking  skills  of  decision-making,  planning  and  producing,  and  predicting — all  skills  related  to 
evaluating  and  creating  in  Table  4. 

For  background  knowledge  on  higher-order  thinking  skills,  trainers  need  to  learn  about 
cognitive  functions,  critical  thinking,  problem-solving,  and  how  these  skills  apply  to  job  tasks. 
The  training  should  include  a  description  of  lower-  and  higher-order  thinking  skills,  the 
difference  between  the  two,  and  should  cover  the  content  and  concepts  contained  in  Table  1  of 
this  report. 

Trainers  will  also  need  to  learn  about  adult  learning  concepts  and  theories,  particularly 
how  they  are  applied  to  promote  critical  thinking.  This  training  should  focus  on  how  adults  learn 
new  concepts,  skills,  and  information.  Training  should  include  discussions  on  instructional 
methods  such  as  collaborative  learning,  discovery  learning,  problem-solving,  discussions,  and 
practical  exercises  that  promote  knowledge  transfer.  Additional  information  should  be  provided 
on  questioning  techniques,  effective  use  of  learner  mistakes,  time  management,  verbal  and  non¬ 
verbal  communication,  and  job  application  relationships. 

Patriot  Battery  unit  trainers  will  require  training  on  how  to  train  decision-making, 
planning  and  producing,  and  predicting.  Much  of  these  skills  can  be  incorporated  into  training 
for  the  decision-making  process,  as  decision-making  requires  an  awareness  of  the  situation 
(analyzing  and  evaluating)  as  well  as  anticipating  effects  of  the  decision  after  it  has  been  made 
(planning  and  producing,  and  predicting).  This  training  would  focus  on  how  the  decision¬ 
making  process  works  so  they  can  implement  instructional  methods  to  foster  its  development, 
incorporating  these  related  cognitive  skills.  At  a  minimum,  this  training  should  consist  of  an 
overall  description  of  the  decision-making  process  in  adults,  the  identification  of  the  components 
involved  in  decision-making,  an  understanding  of  the  internal  and  external  factors  involved  in 
making  decisions,  and  a  general  discussion  of  the  interaction  of  the  factors  involved.  The 
training  should  also  address  how  contextual  factors  can  be  manipulated  to  elicit  problem-solving, 
critical  thinking,  and  decision-making  skills  in  learners. 

Finally,  to  provide  additional  development  of  planning  and  predicting  skills  for 
crewmembers  conducting  ECS/ICC  operations,  trainers  will  need  to  learn  to  create  problem¬ 
solving  opportunities  that  will  drive  crews  toward  having  to  prioritize  actions,  predict  future 
contingencies,  and  plan  for  known  and  possible  future  actions.  While  necessary  and  vitally 
important,  planning  and  predicting  are  complicating  actions  that  require  attention  without 
compromising  immediate  actions  and  operations.  As  a  vital  part  of  operations,  unit  trainers  must 
be  prepared  to  include  planning  and  predicting  in  their  training  program. 
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Step  6:  Potential  Training  Solutions 

A  variety  of  training  solutions  could  address  the  identified  training  gaps.  Considerations 
for  filling  those  shortfalls  may  include  but  are  not  limited  to: 

•  Participation  in  available  TRADOC  Army  instructor  training  courses, 

•  Training  developed  and/or  offered  at  local  or  other  training  installations, 

•  Training  offered  by  civilian  agencies, 

•  Army  and/or  civilian  online  training  courses, 

•  Army  and/or  civilian  resident  and  computer-based  training,  and 

•  Development  of  new  training  to  meet  the  specific  training  needs. 

The  feasibility  of  training  solutions  should  be  judged  by  the  trade-off  considerations  for 
the  resources  needed  to  implement  them.  Resources  may  include  the  availability  of  the  training 
programs,  time  away  from  duties,  funding,  and  equipment.  After  an  analysis  of  potential  training 
solutions  coupled  with  the  availability  of  resources  to  implement  them,  a  determination  can  be 
made  as  to  how  the  training  shortfalls  will  be  overcome. 

Techniques  to  Enhance  Higher- Order  Thinking  Skills  Training 

This  research  sought  to  determine  what  instructors  need  to  know  in  order  to  train  higher- 
order  thinking  skills  for  complex  cognitive  tasks  such  as  decision-making  and  problem-solving. 
The  specific  group  of  instructors/trainers  we  studied  were  specific  to  ADA  Patriot  related  MOSs, 
such  as  the  Master  Gunner’s  Course  at  the  ADA  Schoolhouse.  Many  of  the  recommendations 
we  make  based  on  our  interviews  with  these  instructors  as  well  as  more  general  observations  of 
Army  training  are  intended  to  be  generalizable.  The  training  needs  analysis  found  that  overall 
current  training  addresses  lower-order  thinking  skills  related  to  remembering,  analyzing,  and 
applying  information  due  to  an  emphasis  on  mastering  tactics  and  technology  related  to  the 
Patriot  system.  Complementary  instructor  training  could  support  development  of  learners’  skills 
related  to  higher-order  thinking  skills  by  incorporating  additional  strategies,  methods,  and 
techniques  from  adult  learning  theory.  In  this  final  section  of  the  report,  we  identify  and 
summarize  particular  strategies,  methods,  and  techniques  that  could  be  incorporated  in  Army 
instructor  training  generally,  and  Patriot  training  specifically,  to  prepare  instructors/trainers  for 
developing  the  higher-order  thinking  skills  of  Army  learners. 

Instructional  Approaches  to  Train  Higher- Order  Thinking  Skills 

An  instructional  approach  combines  both  the  theory  and  practice  of  instruction  as  it  is 
applied  to  address  various  learning  objectives  (Kolb,  1984).  It  consists  of  particular  strategies, 
methods,  and  techniques.  An  instructional  strategy  is  the  type  of  instruction  envisioned  for 
learning  activities,  focusing  on  the  teaching  process  and  its  associated  methods  for 
implementation.  An  instructional  method  is  the  way  in  which  learning  content  is  presented. 
Finally,  instructional  techniques  are  the  tools  used  to  target  learning  objectives  (Department  of 
Education  and  Advanced  Learning,  Government  of  Manitoba,  Canada,  2015). 
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Instructional  approaches  are  tailored  to  target  the  training  needs  of  a  particular  audience 
and  training  content  in  order  to  create  a  desirable  learning  environment.  In  selecting  an 
instructional  approach,  Army  trainers  should  consider  how  to  combine  strategies,  methods,  and 
techniques  to  best  support  students  in  achieving  the  specified  learning  objectives.  In  doing  so. 
Army  instructors/trainers  should  consider  the  advantages  and  limitations  of  different  approaches 
(Department  of  Education  and  Advanced  Learning,  Government  of  Manitoba,  Canada,  2015). 

Instructional  Strategies 

Having  facility  in  a  variety  of  instructional  strategies  will  enable  Army  trainers  to  design 
learning  environments  that  help  Soldiers  focus  their  attention  and  organize  their  learning  material 
for  better  understanding.  A  combination  of  strategies  may  be  the  most  appropriate  approach  to 
training  higher-order  thinking  skills  (Ministry  of  Education,  Government  of  Saskatchewan, 
Canada,  1991).  Keesee  (2014)  described  instructional  strategies  in  the  following  way: 

•  Direct  Instruction  is  teacher-directed  and  deductive,  meaning  that  instruction  moves 
from  the  general  (presenting  rules,  principles,  or  laws)  to  specific  examples  or  activities. 

It  includes  methods  like  drill  and  practice,  explicit  teaching  or  lectures.  This  strategy  is 
common  in  Patriot  training  and  in  the  Army,  particularly  for  Soldiers  who  are  early  in 
their  careers.  Eor  Patriot,  direct  instruction  could  include  training  focused  on  the  steps 
required  to  emplace  the  system  and  make  it  operational. 

•  Indirect  Instruction  is  more  student-centered,  taking  into  account  students’  interests. 

The  instructor/trainer’s  role  is  more  of  a  facilitator.  Examples  of  this  method  of 
instruction  would  include  exploring  case  studies,  guided  discussion,  or  collaborative 
problem-solving.  This  strategy  is  often  used  with  Soldiers  as  they  become  more 
advanced  and  can  rely  on  their  own  military  experiences  when  learning  new  information. 
Eor  Patriot,  indirect  instruction  could  focus  on  topics  related  to  how  to  prioritize  multiple 
threats,  or  fix-or-fight  decisions. 

•  Experiential  Learning  is  inductive,  learner  centered,  and  activity  oriented.  It  focuses 
more  on  the  process  than  on  the  product  and  utilizes  methods  such  as  field  trips, 
experiments,  and  role-playing.  This  strategy  can  be  appropriate  for  Soldiers  who  have  a 
variety  of  backgrounds  and  personal/professional  experiences.  Experiential  learning 
methods,  such  as  role-playing,  could  be  used  to  assist  Patriot  crews  in  working  more 
together  more  effectively  as  well  as  to  learn  how  to  communicate  with  units  in  the  chain 
of  command  beyond  the  immediate  crewmembers. 

•  Independent  Study  methods  include  computer-assisted  instruction,  reports,  or  homework. 
As  the  name  suggests,  it  is  work  that  the  student  does  independently,  but  which  may  also 
be  done  in  conjunction  with  other  students.  Again,  this  strategy  may  be  appropriate  for 
Soldiers  who  have  a  variety  of  backgrounds  and  personal/professional  experiences. 

Patriot  crewmembers  may  collect  and  read  the  technical  manuals  and  other  literature 
relevant  to  their  system. 

•  Interactive  Instruction  relies  on  discussion  and  sharing  among  students  and  uses  such 
methods  as  debates,  tutorial  groups,  and  brainstorming.  This  strategy  may  be  most 
appropriate  for  Soldiers  who  have  gained  some  knowledge  and  experience;  however,  it 
can  be  useful  at  all  levels  as  a  culminating  exercise  following  initial  instructional  and 
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study.  In  Patriot,  simulated  air  battles  followed  by  in-depth  after  action  reviews  and 

discussions  would  be  an  example  of  interactive  instruction. 

For  Army  instructors,  some  instructional  strategies  would  seem  more  suited  to  training 
higher-order  thinking  skills.  Indirect  or  interactive  instruction  makes  use  of  learners’  interaction 
and  collaboration  to  develop  knowledge  and  skills  on  their  own.  Given  that  many  Army  learners 
prefer  learning  in  a  hands-on  and  collaborative  context,  indirect  or  interactive  strategies  may  be 
beneficial  for  training  higher-order  thinking  skills  (Graves,  Rauchfuss,  &  Wisecarver,  2012). 
These  approaches  may  be  particularly  effective  when  followed  by  a  carefully-considered  AAR  to 
provide  Soldiers  with  feedback  on  their  performance.  For  Patriot,  scenarios  presenting  the  10 
crew  tasks  listed  in  Table  4  could  be  addressed  in  training  to  elicit  higher-order  thinking  skills. 
Instructors  could  then  guide  discussion  about  crewmembers  performance  toward  issues  involving 
higher-order  thinking  skills  and  techniques  to  improve  analyzing  and  evaluating  information,  and 
creating  effective  solutions  to  problems  encountered  in  Patriot  specific  contexts. 

Instructional  Methods 

Instructional  methods  are  the  ways  in  which  the  learning  content  is  presented.  A  number 
of  factors  guide  the  selection  of  instructional  methods,  for  instance,  the  developmental  level  of 
students,  teaching  goals  or  objectives,  learning  content,  and  environmental  aspects  such  as  time, 
physical  setting,  and  available  resources.  Most  times,  a  single  method  cannot  meet  all  learning 
objectives  or  accommodate  all  learning  needs.  It  is  for  this  reason  that  Army  instructors/trainers 
need  to  develop  a  repertoire  of  methods  from  which  to  construct  their  approach  to  training 
(Petrina,  2007).  As  noted,  what  often  happens  in  Army  training  is  a  focus  on  repetition  that 
tends  to  emphasize  cognitive  skills  such  as  remembering,  understanding,  and  applying. 

Changing  up  instructional  methods  can  encourage  the  development  of  a  broader  array  of 
cognitive  skills  to  include  those  critical  higher-order  thinking  skills  outlined  in  Table  4,  i.e., 
analyzing,  evaluating,  and  creating. 

Existing  training  content  can  be  a  vehicle  to  train  higher-order  thinking  skills.  It  comes 
down  to  how  an  Army  instructor  chooses  to  train  content  that  may  elicit  and  develop  higher- 
order  thinking  skills.  Instructors  must  purposefully  design  those  opportunities  through  the 
strategies  and  methods  they  select. 

Army  instructors  should  be  mindful  that  they  always  are  training  adults  and  should  be 
knowledgeable  about  adult  learning  needs.  Research  has  consistently  supported  Knowles  (1984) 
observations  that  adults  learn  best  (a)  if  they  know  why  they  are  learning  something,  (b)  when 
they  learn  through  experience,  (c)  when  the  topic  is  relevant  and  immediately  applicable  to  them, 
and  (d)  when  they  can  view  learning  as  an  opportunity  to  solve  real-world  problems.  The  two 
instructional  strategies  that  most  encourage  active  participation  (interactive  and  indirect 
instruction)  are  consistent  with  Knowles’  (1984)  observations  about  adult  learners.  These 
instructional  strategies  also  tend  to  be  well  suited  to  eliciting  and  developing  higher-order 
thinking  skills  through  discussions  between  learners. 

As  adult  learners.  Soldiers  benefit  from  each  other’s  experience  and  knowledge.  Open 
discussions  present  an  opportunity  for  Soldiers  to  become  active  in  learning  and  to  develop 
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personal  contacts  with  others,  contacts  that  may  support  ongoing  professional  development. 
Moreover,  good  discussions  can  be  challenging  to  Soldiers’  preexisting  beliefs  and  perspectives, 
pushing  them  to  formulate  new  ideas  and  applications  for  what  they  know  (see  University  of 
California,  Los  Angeles,  2011).  The  instructor/trainer  acts  as  a  moderator,  steering  the 
discussion  to  encourage  in-depth  analysis  of  the  content  domain.  Discussions  should  have  an 
objective,  such  as  coming  up  with  a  solution  to  a  problem,  completing  a  task,  or  reaching  a 
consensus  (Nihill  &  Stallings,  2014).  Ultimately,  discussions  elicit  higher-order  thinking  skills 
related  to  comparing  and  contrasting  points  of  view,  solutions,  and  reasoning. 

Army  instructors/trainers  may  employ  a  variety  of  techniques  to  elicit  and  develop 
higher-order  thinking  skills  when  facilitating  discussions.  Often  during  discussions,  learners  will 
move  quickly  to  proposing  solutions.  The  facilitator  should  seek  to  extend  the  problem-solving 
part  of  the  discussion  in  order  to  develop  the  discussion.  Other  ways  to  develop  discussions 
include  shifting  context  and/or  points  of  view,  eliciting  discussions  of  costs/benefits,  shifting 
levels  of  abstraction  to  further  clarify  and  develop  ideas,  asking  ‘what  if...’  questions,  and 
highlighting  points  of  disagreement  in  the  discussion  (see  Indiana  University,  2012,  for 
additional  discussion  of  facilitation  techniques).  The  following  are  techniques  that  may  be 
particularly  applicable  in  an  Army  training  setting. 

Cooperative  Learning  Groups.  Cooperative,  or  collaborative,  learning  is  a  method  in 
which  students  with  different  abilities  form  teams  to  conduct  learning  activities  (Gokhale,  1995). 
Team  members  are  responsible  for  both  their  own  learning  and  that  of  their  collaborators  (U.S. 
Department  of  Education,  1992).  Cooperative  learning  groups  are  similar  to  discussions  in  that 
group  members  offer  their  own  point(s)  of  view  while  also  considering  and  analyzing  those  of 
others  (Westberg  &  Jason,  1996).  Moreover,  group  members  must  work  together  to  analyze 
material,  make  decisions,  solve  problems,  etc.  (Brown  &  Lara,  2007). 

Problem-Based  Learning.  Problem-solving  is  an  activity  in  which  learners  take  what 
they  know  and  apply  it  to  discover  what  they  do  not  know.  Problem-based  learning  (PBL) 
encourages  critical  thinking  and  problem-solving  skills  and  involves  participants  confronting 
contextualized,  ill-structured  problems  and  finding  solutions  by  testing  hypotheses  for 
satisfactory  answers  (Major,  1998).  In  PBL,  the  learner  is  given  details  about  a  problem  and  the 
objectives  and  challenges  are  outlined.  Additional  data  is  given  in  which  the  learner  then  uses  to 
analyze  the  situation,  determine  what  happened,  and  make  recommendations. ' 

Problem-solving  activities  “develop  student  ability  to  solve  problems  using  knowledge, 
concepts,  and  skills  relevant  to  a  course”  (McKeachie,  2002).  Moreover,  problem-solving 
provides  real-world  context  for  learning,  which  may  increase  transfer  of  knowledge  from  the 
classroom  to  the  field.  Problem-solving  methods  are  student-centered,  with  the  trainer  offering 
guidance  as  needed.  Some  useful  techniques  for  utilizing  PBL  in  training  are:  (a)  model  a  usable 
problem-solving  method,  (b)  teach  problem-solving  skills  in  the  context  they  will  be  used,  not  as 
an  abstract  skill,  (c)  help  students  understand  the  what  and  why  of  a  problem  first  before 
determining  how,  (d)  budget  time  to  define  the  problem  and  understand  the  goal,  (e)  ask  open- 


'  Levine  (1994)  provides  a  lengthy  discussion  of  how  PBL  may  be  used  to  support  development  of  higher- 
order  thinking  skills. 
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ended/what  if  questions,  and  (f)  link  students’  errors  to  how  they  may  be  misconceiving  the 
problem  (University  of  Waterloo,  2015). 

Brainstorming.  Brainstorming  is  a  method  that  is  effective  both  on  the  job  and  in 
training  contexts.  When  brainstorming,  learners  offer  potential  solutions  to  a  problem, 
regardless  of  feasibility.  All  input  is  recorded  initially  without  prejudice,  for  subsequent 
analysis.  It  encourages  a  relaxed,  informal  approach  to  problem-solving  and  can  result  in 
thoughts  and  ideas  that  can  be  crafted  into  creative,  original  solutions. 

In  training,  brainstorming  may  be  used  differently  than  in  the  workplace.  Brainstorming 
in  the  training  context  encourages  learners  to  learn  to  think  creatively  and  critically.  It  is  by 
generating  and  then  evaluating  ideas  that  brainstorming  can  develop  critical  thinking  and  other 
higher-order  thinking  skills  (Bacal,  2015).^ 

Case  studies.  Case  studies  are  descriptions  of  unresolved  and  provocative  issues, 
situations,  or  questions  drawn  from  real-world  events  (Crockett  &  Foster,  2005).  Cases 
challenge  adult  learners  to  analyze,  critique,  make  judgments,  speculate,  and  express  opinions 
(Indiana  University,  2012).  Use  of  case  studies  can  result  in  improved  retention,  recall,  and 
transfer  of  learning  to  other  contexts  (McKeachie,  2002). 

To  work  through  a  case  study,  learners  are  assembled  into  groups  to  discuss  it  and 
formulate  solutions  (Brown  &  Lara,  2007).  Case  studies  give  students  practice  in  identifying  the 
parameters  of  a  problem,  and  help  them  to  recognize  and  articulate  positions,  evaluate  courses  of 
action,  and  argue  different  points  of  view  (Crockett  &  Foster,  2005).  Case  studies  vary  in  length, 
detail,  and  use  depending  on  the  case  itself  and  on  the  instructor’s  goals. 

Instructional  Techniques 

Army  trainers  must  learn  to  implement  a  variety  of  instructional  methods  to  foster 
development  of  higher-order  thinking  skills  in  adult  learners.  Instructional  techniques  are 
applicable  across  many  situations  to  promote  critical  thinking  and  learning.  In  accord  with 
Knowles’  (1984)  characteristics  of  adult  learners,  five  complementary  principles  for  adult 
learning  have  been  identified  (Vandenberg,  1998): 

•  Personal  Benefit.  Adult  learners  must  be  able  to  see  the  personal  benefit  of  what  they 
are  learning,  and  how  it  satisfies  a  need  they  have. 

•  Experience.  Adult  learners  are  motivated  to  learn  if  the  learning  involves  them  in 
sharing  what  they  know,  builds  on  what  they  know,  and  validates  their  experience. 

•  Self-Direction.  Adult  learners  are  motivated  to  learn  if  they  can  take  charge  of  their 
learning  and  make  decisions  about  the  content  and  process,  contribute  to  the  learning  of 
their  co-learners,  and  have  some  degree  of  independence  in  the  learning  process. 

•  Application  and  Action.  Adult  learners  are  busy,  practical,  and  learn  by  doing. 

•  Learning  Styles.  Adult  learners  learn  best  when  the  learning  taps  into  a  mix  of  learning 
styles  that  fit  their  preferences. 

^  The  entrepreneur  Richard  Branson  (2012)  has  offered  a  useful  discussion  of  the  brainstorming  process  for 
those  who  would  like  to  apply  the  technique  more  effectively. 
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Creating  an  environment  that  is  interactive,  in  which  learners  take  an  active  part  in  the  learning 
process,  helps  promote  better  learning  in  adults. 

Facilitation.  Facilitation  is  an  adult  learning  technique  that  shifts  the  focus  of  the 
learning  environment  from  teacher-centered  to  student-centered  (U.S.  Department  of  the  Army, 
TRADOC,  201 1).  The  trainer’s  role  is  to  guide  the  learning  activity,  not  direct  it.  Trainers  focus 
on  drawing  out  learners’  ideas  and  experiences.  Effective  facilitators  (a)  begin  by  asking 
questions,  (b)  introduce  information  slowly,  (c)  make  the  content  applicable,  (d)  learn  from  their 
students’  experiences,  (e)  encourage  questions  and  idea  sharing,  (f)  are  clear  and  direct,  (g)  allow 
time  for  students  to  think,  and  (h)  keep  their  own  contributions  brief  (Sunnarborg,  2008). 

Effective  Questioning.  Effective  questioning  is  focused  on  purposefully  structuring  and 
using  questions  to  help  students  learn,  not  to  determine  whether  they  have  learned  (Nihill  & 
Stallings,  2014).  Questions  may  be  targeting  lower-  or  higher-order  thinking  skills.  Questions 
asking  learners  to  state,  list,  name,  or  describe  are  focused  on  lower-order  thinking  skills; 
questions  asking  learners  to  justify,  discuss,  convince,  differentiate,  or  defend  are  focused  on 
higher-order  thinking  skills. 

Job-Application  Relationship.  Adult  learners  tend  to  be  motivated  by  learning 
knowledge  and  skills  that  are  applicable  to  their  personal  or  work  lives.  Trainers  are  responsible 
for  creating  conditions  for  learning  that  highlight  the  applicability  of  what  is  being  taught 
(Keesee,  2011).  Adult  learners  are  motivated  when  they  believe  they  need  to  learn  something 
new  or  different  (Zemke  &  Zemke,  1995).  Presenting  the  learning  content  in  applicable,  real- 
world  settings  helps  stimulate  learning.  Adult  learners’  level  of  motivation  can  be  increased  if 
the  instructor  demonstrates  the  usefulness  of  what  is  being  taught,  creates  a  safe  environment  for 
the  learner,  and  explores  learners’  expectations  about  the  learning  experience  (Goldstein,  2012). 

Knowledge  Transfer.  The  idea  of  knowledge  transfer,  in  simple  terms,  is  when  a  person 
takes  something  learned  in  one  setting  (such  as  in  the  classroom)  and  applies  that  knowledge  to  a 
new  setting  (such  as  a  real-world  activity  or  new  situation)  (Nihill  &  Stallings,  2014).  Eor 
transfer  to  occur,  learners  cannot  simply  memorize  information  but  must  also  understand  what 
they  have  learned  in  order  to  apply  what  they  learned  to  other  situations  (Bransford,  Brown,  & 
Cocking,  2000). 

In  a  learning  environment,  a  trainer  may  reinforce  the  learning  of  new  material  presented 
by  having  learners  apply  the  material  in  a  practical  exercise.  In  addition  to  helping  solidify  the 
new  material,  learners  additionally  practice  the  transfer  of  knowledge  as  they  apply  that  material 
to  new  situations  (Nihill  &  Stallings,  2014).  Darling-Hammond  and  Austin  (2003)  provide  that 
learners  should  be  allowed  to  practice  transfer  in  a  variety  of  contexts. 

Use  of  Mistakes.  Mistakes  made  while  learning  should  be  viewed  as  acceptable  and 
positive  occurrences,  and  part  of  the  learning  process.  Mistakes  are  valuable  tools  to  foster 
alternative  and  critical  thinking.  Environments  where  mistakes  are  viewed  negatively  can  hinder 
learning.  Eearners  that  can  identify  mistakes  and  reflect,  analyze,  and  examine  their  errors  are 
able  to  improve  their  understanding  and  higher-order  thinking  skills  (Nihill  &  Stallings,  2014). 
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Mistakes  made  during  training — if  they  are  addressed  carefully — can  lead  to  fewer  mistakes  in 
practice.  This  may  also  assist  learners  is  being  able  to  better  predict  the  outcomes  of  various 
decisions  they  may  make  and  problem-solving  strategies  they  may  employ. 

Example  Instructional  Approaches 

To  develop  higher-order  thinking  skills,  Army  instructors/trainers  need  to  be  able  to  use 
the  existing  course  content  and  training  objectives  in  ways  that  challenge  learners’  expectations. 
A  well-designed  and  executed  combination  of  instructional  strategies,  methods,  and  techniques 
can  offer  an  instructional  approach  that  develops  higher-order  thinking  skills  in  learners.  The 
following  provides  an  example  of  how  to  revise  a  training  approach  to  better  develop  higher- 
order  thinking  skills. 

Military  Decision-Making  Process  Example.  An  instructor  for  a  U.S.  Army  resident 
school  is  preparing  to  restructure  the  training  approach  for  a  class  on  operational  planning  and 
the  military  decision-making  process  (MDMP).  The  main  objective  of  the  class  is  to  teach  the 
MDMP,  a  seven-step  process,  to  the  personnel  in  the  class.  Inherent  to  the  MDMP  are  analytical 
and  decision-making  steps.  Currently,  the  lesson  plans  include  lecture-based  classes  describing 
the  MDMP  followed  by  a  take  home  practical  exercise  in  which  class  members  apply  the  process 
independently  and  submit  their  write-up  and  conclusions  to  the  instructor.  The  instructor  then 
reviews  and  provides  comments  to  individuals  and  selects  the  best  submission  to  be  used  as  an 
example  to  be  explained  to  the  rest  of  the  class. 

In  the  revised  training  approach,  the  trainer  wants  to  accomplish  more  in  the  class  than 
simply  having  Soldiers  memorize  and  apply  the  steps.  The  trainer’s  objectives  include  having 
the  Soldiers  learn  how  to  critically  apply  the  steps  and  increase  their  analytical  and  decision¬ 
making  skills.  In  the  revised  approach,  the  trainer  plans  to  use  adult  learning  concepts  and 
methodologies  as  a  way  to  increase  learning  and  retention,  and  enhance  higher-order  thinking 
skills.  The  trainer  decides  to  make  wide  use  of  the  interactive  instructional  strategy  with  its 
associated  instructional  methodologies  and  then  apply  adult  learning  techniques  during 
instruction.  To  accomplish  those  goals,  the  trainer  realizes  that  some  direct  instruction  is 
necessary  to  set  foundations  but  also  decides  to  include  the  items  in  Table  5  in  the  design  of  the 
revised  training  approach  to  enhance  higher-order  learning. 
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Table  5 


Characteristics  of  the  Revised  Training  Approach  to  Foster  Higher-Order  Thinking  Skills 
During  Training  in  the  MDMP. 


Methods  and  Techniques 

Benefit 

A  short  video  of  an  Army  staff 
section  applying  the  MDMP  to  a 
new  mission  to  begin  the  class 

Use  of  technology,  visual  application  of  material  to  work  related 
events,  sparks  interest  in  learners,  portrays  why  the  learning 
content  is  necessary 

Organize  learners  into  small 
groups  to  apply  the  steps  of  the 
MDMP  in  a  classroom  setting 
rather  than  each  learner 
individually  on  their  own 

Variety  of  opinions  shared  between  learners,  learners  see  other 
points  of  views,  learners  consider  and  analyze  thoughts  of  others, 
other  views  stimulate  additional  thoughts  in  individual  learners  as 
they  broaden  their  thinking,  learners  promote  learning  in  others, 
increased  attention  to  succeed  as  a  group 

Make  wide  use  of  having 
questions  that  are  asked  during  the 
conduct  of  the  practical  exercise 
to  he  answered  hy  other  students 

Learners  learn  from  each  other,  forces  learners  to  seek  out 
answers  rather  than  simply  being  told  the  answer  helping  to 
increase  retention 

The  selection  of  a  random 
member  from  each  group  will  be 
required  to  explain  their 
conclusions  along  with  the  results 
of  their  analysis  and  justification 
that  led  to  the  decisions  made 

Forces  critical  thinking  from  all  learners,  learners  must  perform 
analysis  of  material  presented  in  order  to  explain  the  justification 
of  decisions 

During  presentation  of  results 
allow  for  other  groups  to  ask 
questions  of  the  presenter 

Learners  learn  from  each  other,  presenters  and  other  learners 
involved  in  critical  thinking 

Facilitate  comments  regarding  the 
presenter’s  analysis,  justification, 
and  decisions  away  from  criticism 
and  towards  critical  thinking  by 
allowing  for  and  offering 
alternative  solutions  to  consider 

Makes  positive  use  of  less  than  optimal  solutions,  allows  for 
critical  reevaluation  of  decisions  and  conclusions 

Other  examples.  Table  6  provides  additional  examples  of  methods  of  how  higher-order 
thinking  skills  may  be  fostered  during  training. 
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Table  6 


Potential  Ways  to  Foster  Higher-Order  Thinking  Skills  During  Training 


Training  example 

Potential  methods 

Benefit 

Training  combat 
medics  how  to  identify 
and  treat  an  injury 

Discussion/problem  solving  -  Use  the 
discussion  method  in  conjunction  with 
problem  solving  to  diagnose  a  hypothetical 
injury.  Instructor  facilitates  to  steer 
discussion  and  generate  critical  thinking  and 
analysis. 

Students  exercise  higher- 
order  thinking  skills  by 
combining  factual 
knowledge  with  analysis  and 
critical  thinking  to  develop 
alternatives  and  make 
decisions. 

Training  Air  Defense 
system  maintainers  in 
an  electrical 
maintenance  class  how 
to  determine  the  cause 
of  equipment  failure 

Problem  solving/problem  based  learning  -  A 
simulated  event  is  designed  to  present 
students  with  real  world  equipment  failures 
in  which  students  must  determine  the 
location  and/or  cause  of  the  issue  and  the 
solution  for  fixing  it. 

PBL  is  active  as  opposed  to 
passive.  Allows  students  to 
analyze  a  problem  and  apply 
what  they  know.  Analysis  of 
input  allows  for  discovery  of 
what  students  don’t  know  or 
need  to  know. 

Training  firefighters  on 
how  to  best  attack 
different  types  of  fires 

Brainstorming  -  Firefighters  offer  ideas  on 
how  to  best  attack  a  problem  or  unique  fires. 
All  opinions  are  recorded  and  then  later 
discussed  and  analyzed  for  pros  and  cons. 

Stimulates  alternative 
thinking,  critical  thinking, 
the  sharing  of  ideas  and 
learning  from  learners. 

Patriot  Specific  Instructional  Approaches.  Essential  to  any  training  that  seeks  to 
effectively  elicit  and  develop  higher-order  thinking  skills  is  an  instructor/trainers  ability  to  alter 
the  learning  environment.  Given  the  very  hands-on  and  practice  oriented  approach  in  Patriot 
training,  the  learning  environment  may  be  manipulated  in  a  number  of  ways.  In  simulations  with 
Patriot  crewmembers,  for  example,  instructors/trainers  can  alter  scenarios  by  incorporating 
unexpected  elements.  Examples  of  critical  tasks  that  can  be  incorporated  into  training  to 
challenge  learners  higher-order  thinking  skills  include:  fix-or-fight  decisions,  establishing  or 
altering  task  priorities,  classification  of  tracks,  requesting  assistance,  predicting  future  track 
actions,  predicting  track  separation,  decisions  to  launch,  decisions  to  execute  manual  or 
automatic  engagements,  and  when  to  execute  transfer  of  control. 

Army  instructors/trainers  cannot  directly  control  the  internal  cognitive  processes  of  their 
students.  They  can,  however,  elicit  and  develop  these  skills  by  manipulating  the  external  context 
in  which  training  is  taking  place.  Aspects  of  the  learning  environment  that  can  be  altered  to  good 
effect  include  the  information  available  to  students,  the  requirements  of  the  task,  factors  that 
affect  the  decision-makers’  mental  status  (e.g.,  imposing  time  constraints  to  induce  stress), 
factors  affecting  the  decision-makers’  physical  status  (e.g.,  conducting  the  exercise  under  mock 
CBRN  conditions),  and  changing  the  operating  environment  (e.g.,  introduce  an  unexpected 
weapons  system). 

Manipulating  Information.  Information  available  to  Patriot  crews  can  be  manipulated 
in  a  variety  of  ways  to  elicit  higher-order  thinking  skills.  Eor  instance,  information  from 
different  sources  can  be  combined  to  augment  or  contradict  particular  interpretations  of  what  is 
happening  in  the  battlespace.  Moreover,  information  flow  can  be  increased  or  decreased  during 
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a  simulated  air  battle,  track  data  may  be  altered,  or  the  data  feed  may  be  suddenly  cut.  Each  of 
these  events  would  be  unexpected  and  could  be  used  to  elicit  and  practice  higher-order  thinking 
skills. 


Manipulating  the  Complexity  of  Tasks.  Patriot  instructor/trainers  can  manipulate  the 
complexity  of  tasks  to  which  learners  are  exposed.  This  can  be  accomplished  by  increasing  or 
decreasing  the  number  of  threats  the  learners  must  track  as  well  as  introducing  additional  factors 
that  affect  decision  making  such  as  scheduling  launcher  reloads  or  dealing  with  maintenance 
faults.  The  information  being  supplied  to  a  learner  may  also  evolve  in  order  to  create  a  situation 
in  which  crewmembers  have  to  reevaluate  their  assumptions.  This  could  be  accomplished 
through  updating  data  on  a  track,  changing  data  for  a  track  (such  as  splitting  a  track), 
reclassifying  a  known  track,  providing  data  that  has  multiple  interpretations,  or  sending 
electronic  message  updates. 

Manipulating  Physical  and  Mental  Effects.  Operating  under  conditions  of  fatigue  or 
physical  stress  can  challenge  learners’  cognitive  processes  and  decision-making  abilities.  Patriot 
instructor/trainers  can  create  conditions  of  physical  stress  by  altering  the  physical  environment, 
such  as  increasing  ambient  noise  or  altering  lighting  conditions.  Other  ways  to  manipulate  this 
aspect  of  the  learning  environment  could  include  conducting  a  no  notice  training  event,  or 
extending  a  training  event  beyond  the  regular,  and  expected,  time  period.  Manipulating  mental 
effects  can  be  accomplished  by  exposing  learners  to  ambiguous  situations  with  a  constraint  on 
their  time  to  act,  such  as  creating  target  saturation  on  scope  with  additional  hostile  tracks  from 
the  flank,  or  a  situation  in  which  learners  are  forced  to  prioritize  their  responses  to  multiple 
threats. 


Manipulating  the  Operating  Environment.  Changes  to  the  operating  environment  can 
also  be  effective  in  eliciting  complex  cognitive  skills.  For  instance,  a  Patriot  crew  could  be 
exposed  to  changes  in  the  assistance  available  from  outside  the  Engagement  Control  Station.  For 
instance,  this  situation  could  come  about  due  to  communications  being  cut  with  command  and 
control  elements.  Also,  a  higher  level  supervisor  could  be  present  during  an  exercise,  asking 
crewmembers  questions  or  giving  orders,  or  a  crewmember  could  be  placed  out  of  action  during 
a  simulated  air  battle.  Finally,  changes  to  standard  procedures,  methods,  or  mandated  guidance 
immediately  prior  to  or  during  training  can  force  crews  to  engage  in  complex  decision-making 
and  problem-solving  tasks. 

When  conducting  Patriot  training,  each  of  these  aspects  of  the  training  environment  can 
be  manipulated  to  elicit  higher-order  thinking  skills.  An  effective  discussion  and  after  action 
review  would  necessarily  follow  to  allow  crewmembers  to  make  sense  of  how  they  responded  to 
the  challenges  they  were  presented  with,  and  identify  which  responses  were  effective  and  which 
were  not.  It  is  during  this  review  and  discussion  that  instructors/trainers  can  incorporate 
discussions  of  higher-order  thinking  skills  and  provide  feedback  essential  to  improving 
crewmembers  decision-making  and  problem-solving  processes. 
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Conclusions 


General  Higher- Order  Thinking  Skills  Training  Recommendations 

Developing  higher-order  thinking  skills  in  training  should  be  the  result  of  conscious 
effort.  By  increasing  critical  thinking  skills,  students  can  improve  their  overall  problem-solving 
abilities,  decision-making  abilities,  and  general  performance.  Research  has  suggested  a  number 
of  positive  outcomes  from  developing  higher-order  thinking  skills  (Foundation  for  Critical 
Thinking,  2014).  Higher-order  thinking  skills  training  can  develop  learners  who: 

•  raise  vital  questions  and  problems,  formulating  them  clearly  and  precisely; 

•  gather  and  assess  relevant  information,  using  abstract  ideas  to  interpret  it  effectively; 

•  come  to  well-reasoned  conclusions  and  solutions,  testing  them  against  relevant  criteria 

and  standards; 

•  think  open-mindedly  within  alternative  systems  of  thought,  recognizing  and  assessing,  as 

need  be,  their  assumptions,  implications,  and  practical  consequences;  and 

•  communicate  effectively  with  others  in  figuring  out  solutions  to  complex  problems. 

To  develop  higher-order  thinking  skills  in  training.  Army  trainers  must  be  prepared  with 
the  instructional  skills  and  knowledge  to  accomplish  that  goal.  The  TNA  designed  for  this 
research  focused  on  analyzing  the  performance  of  trainers  in  developing  higher-order  thinking 
skills  during  training.  While  the  focus  of  the  research  concerned  ADA  unit  trainers,  the  results 
of  the  TNA  and  the  recommended  strategies,  methods,  and  techniques  covered  in  the  discussion 
may  be  applied  by  a  broad  range  of  Army  trainers. 

Once  trainers  are  equipped  with  the  knowledge  and  tools  for  developing  higher-order 
thinking  skills  during  training,  and  they  have  decided  what  specific  skills  they  wish  to  focus  on, 
they  will  need  to  select  the  instructional  methods  they  wish  to  use  from  among  the  many 
available.  Creative  trainers  can  find  interesting  and  engaging  ways  to  implement  the  various 
methods  to  help  accomplish  the  objective  of  fostering  higher-order  thinking.  Appendix  B  shows 
examples  of  how  some  of  the  methods  may  be  used  to  foster  specific  higher-order  thinking  skills. 

ADA  Patriot  Missile  Crew  Training 

This  research  focused  on  what  ADA  Patriot  missile  crew  unit  trainers  need  to  know  in 
order  to  develop  higher-order  thinking  skills  among  the  Soldiers  they  train.  Given  the  time- 
sensitive  and  often  ambiguous  situations  in  which  ADA  crews  operate,  training  in  higher-order 
thinking  skills  may  help  to  reduce  the  time  required  for  engagement  decisions  and  increase  the 
effectiveness  of  crews  under  high-stress  situations. 

While  many  of  the  recommendations  may  be  useful  to  ADA  School  instructors,  the  focus 
of  the  TNA  was  unit  trainers.  ADA  School  instructors  primarily  teach  personnel  how  to 
initialize  the  ECS/ICC  system  and  subsequently  introduce  them  to  van  operations.  Unit  trainers, 
predominately  the  SO  and  the  Battery  trainer  in  the  Patriot  missile  battery,  are  responsible  for  the 
majority  of  advanced  crew  operational  training  where  decision-making  actions  occur.  Further, 
research  revealed  that  in  contrast  to  schoolhouse  instructors  who  receive  trainer  training  through 
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courses  such  as  ABIC,  unit  trainers  most  often  receive  no  formal  preparation  for  their  duties  as 
trainers. 

This  research  determined  that  Patriot  missile  battery  unit  trainers  must  have  the  ability  to 
determine  and  implement  applicable  adult  learning  techniques  and  methodologies  in  the  learning 
environment  in  order  to  foster  the  cognitive  processes  involved  in  decision-making.  This  ability 
is  necessary  in  order  to  promote  improvement  of  decision-making  skills  in  learners.  To 
accomplish  that  mission,  trainers  require  the  appropriate  knowledge  and  trainer  preparation 
needed  for  implementation. 

This  report  is  the  first  in  a  series  of  two  reports  focused  on  what  Army  instructors/trainers 
need  to  know  in  order  to  develop  higher-order  thinking  skills.  The  second  report  will  present  an 
8-hour  block  of  instruction  that  was  developed  based  on  the  results  of  the  literature  review  and 
training  needs  analysis  presented  here.  In  addition,  the  follow-on  report  will  present  the  results 
of  validation  research  conducted  using  the  8-hour  block  of  instruction  with  ADA  Schoolhouse 
instructors  and  unit  trainers. 
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List  of  Acronyms 


AAR 

After- Action  Review 

ABIC 

Army  Basic  Instructor  Course 

ADA 

Air  Defense  Artillery 

AMD 

Air  and  Missile  Defense 

BDE 

Brigade 

Btry 

Battery 

CW(0) 

Chief  Warrant  (Officer) 

ECS 

Engagement  Control  Station 

EMMO 

Electronic  Missile  Maintenance  Officer 

HOTS 

Higher-Order  Thinking  Skills 

ICC 

Information  Coordination  Central 

MDMP 

Military  Decision-Making  Process 

MOS 

Military  Occupational  Specialty 

MTOE 

Modified  Table  of  Organization  and  Equipment 

NCO 

Noncommissioned  Officer 

Patriot 

Phased- Array  Radar  Intercept  to  Track  on  Target 

PBE 

Problem-Based  Eeaming 

SME 

Subject  Matter  Expert 

SO 

Standardization  Officer 

SPEAR 

Standardized  Patriot  Evaluation  and  Assessment  Reporting 

TNA 

Training  Needs  Analysis 

TRADOC 

Training  and  Doctrine  Command 

USAADASCH 

U.S.  Army  Air  Defense  Artillery  School 

WOCC 

Warrant  Officer  Career  College 

woes 

Warrant  Officer  Candidate  School 

WOIEE 

Warrant  Officer  Intermediate  Eevel  Education 
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APPENDIX  A 
Decision-Making  Diagrams 
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r 


Colketion 
(Input  6ath«ring) 


Input  sources 

•  May  bs  numerous  or  few 

•  May  be  produced  by  or  originate 
from  different  sources 

•  May  be  delivered,  transmitted,  or 
presented  through  various  means: 

o  Verbally 
o  Written 
o  Sensory 

>  Audible  sound 

>  Visual  observation 

>  Other  senses 
o  Electronically 

o  Combination 
o  Other 

•  May  provide  continual  or  evolving 
information 


Staff/orgonizationol  support  may 
filter  information  to  determine 
what  available  information  is 
passed  on  to  the  decision  maker 
Filtering  may  be  accomplished 
manually  and/or  electronically 


Staff /Organization 
Input  Processing 


Staff/organizational  support  may 
investigate,  verify,  categorize, 
combine,  or  reduce  information  to 
assist  the  decision  maker 


Figure  A-1.  Factors  affecting  decision-making  -  Expanded  collection  section. 
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Effects  From 
Intemol  Foctors 

(effects  due  to  the  personol  ehoroeteristies  of  a  person) 


Personality  Traits 

■  Cognitive  abilities 

•  Risk  attitude 

•  Ambiguity  intolerance 

■  Self  confidence 

•  Values 

■  Stress  tolerance 

•  Willingness  to  assume 
responsibility 


Experience  &  Skills 

•  Technical  skills 

•  Leadership  skills 

•  Training 

•  Post  experiences 

•  Experience  in  making 
type  of  decision 


AAotives 

•  Personol 

•  Level  of 


interest 
commitment 


Heuristics  Biases 

•  Availability 

•  Representativeness 

•  Anchoring 

•  Affect 


Figure  A-2.  Factors  affecting  decision-making  -  Expanded  effects  from  internal  factors  section. 
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Input  Characteristics 

■  Physical  characteristics  of  input  sources 
o  ^fects  of  presentation  of  information 
o  IT  management  -  technology  input/assistance 
o  Number  of  input  sources 

•  Quality,  quantity,  and  frequency  of  inputs 
o  Credibility  of  information  from  each  source 
o  Conflicting  information  from  sources 
o  Completeness  of  information  from  each  source 
o  Amount  of  information  received  from  each  source 
o  Frequency  of  information  received  from  each  source 

■  Evolution  of  information 
o  Situation  dynamics 

o  Information  dynamics  from  a  singe  source 
o  Dynamics  of  input  sources 


Task  Characteristics 

•  Complexity  of  the  tosk 

•  Relevance  of  decision 
o  High/low  value 

o  Associated  threat  involved 


Effects  From 
Externol  Foetors 

(effects  due  to  influences  outside  of 
personol  chorocteristics) 


Mental  Effects 

•  Constrained  time  stress 

•  Stress 

•  Performance  pressure 

•  High  work  load 

•  Information  overload 

o  Sop  between  information  received  and 
ability  to  adopt 

o  Receiving  nnore  information  thon  con  be 
adequately  processed 
o  Loss  of  information  due  to  inability  to 
manage 


I  Intcfyctqtion 
I  Port  2 


Physical  Effects 

•  Fatigue 

•  Thirst 

•  Noise 


Operating  Environment 

•  Staff/organization  input  assistance 

o  Size  and  ability  of  others  to  reduce 
information/provide  assistance 
o  Credibility  of  information  provided  by 
others 

•  Standard  procedures  or  methods 

•  Mandated  guidance 

_ y 


Figure  A-3.  Factors  affecting  decision-making  -  Expanded  effects  from  external  factors  section. 
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Preducfion 


excision  maker  analyzes  information 
received,  considers  possible  explanations 
for  individual  and  collective  data, 
evaluates  the  drivers  behind  the 
interpretations  derived,  and  develops 
alternative  responses 

Znterpretetion 

^  Part  8  


Judgment 

Decision  maker  weighs  various 
alternatives,  possible  actions 
or  responses  to  conclusions 
reached  based  on  the  analysis 
and  evaluation 

_ 


Decision  maker  selects 
the  course  of  action  or 
response  most  justifiable 
from  an  understanding 
derived  from  the 
information  received 

-ri 


4 

4 


Decision 


Figure  A-4.  Factors  affecting  decision-making  -  Expanded  analysis  and  evaluation,  processing, 
and  productions  sections. 
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APPENDIX  B 

Methods  for  Fostering  Higher-Order  Thinking  Skills 
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Table  B-1 

Methods  for  Fostering  Higher-Order  Thinking  Skills. 


HOTS 

Instructional 

Method 

Description 

Purpose 

Example 

Analyzing, 

deciding, 

planning 

and 

producing 

Cooperative 

learning 

Small  teams  comprised 
of  students  of  different 
ability  levels  use  a 
variety  of  learning 
activities  to  improve 
their  understanding  of  a 
subject. 

To  have  students  apply 
theoretical  knowledge  to 
real-world  problems,  or 
demonstrate  decision¬ 
making  or  problem-solving 
skills  similar  to  those  made 
by  professionals  in  the  field. 

Students  are  divided  into  small  groups 
of  five  to  six  people  each.  A  problem 
situation  is  given  to  all  groups  that 
require  collection  of  various  types  of 
data,  analysis  of  the  data,  development 
of  alternatives,  selection  of  an 
alternative,  and  production  of  written 
findings.  Have  each  group  share  its 
solution  and  justification  with  other 
groups. 

Classifying, 

analyzing, 

deciding 

groups 

Students  are  divided  into  small  groups 
of  five  to  six  people  and  each  group  is 
given  20  objects.  Have  each  group 
establish  a  rationale  for  its 
classification,  choose  five 
characteristics  deemed  most  crucial  to 
classification,  and  subsequently 
classify  the  objects  and  justify  the 
decisions. 
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HOTS 

Instructional 

Method 

Description 

Purpose 

Example 

Determining 
cause  and 
effect 

Reflective 

discussion 

A  facilitated  discussion 
where  participants  are 
encouraged  to  think 
and  talk  about  what 
they  have  observed, 
heard  or  read 
(McCambridge,  n.d.). 

To  challenge  students' 
thinking  by  inviting  them  to 
interpret,  infer,  summarize, 
form  conclusions  and 
evaluate  selections.  To  help 
create  analytical  thinkers 
and  minds  that  can  work 
through  complex  problems. 

Show  a  video  or  have  students  read  a 
synopsis  of  an  event  with  unpredicted 
catastrophic  consequences.  Have  the 
students  first  discuss  the  effect  of  the 
event  (what  happened);  then  the  cause 
of  the  effect  (the  reason  that  it 
happened);  and  finally  the  cause  and 
effect  (an  explanation  of  why  it 
happened).  Subsequently  students  may 
offer  suggestions  of  how  the  effect 
could  have  been  prevented. 

Critiquing, 

analyzing, 

deciding 

Role  playing 

A  method  of  learning 
by  exploring  issues  in 
complex  social 
situations  through 
hypothetical  situations 
that  allows  students  to 
practice  their  new 
knowledge  before 
application  in  the  real 
world  (Schumann, 

2002). 

To  have  student 
immediately  apply  content 
in  a  relevant,  real-world 
context.  Students  engage  in 
higher-order  thinking  by 
thinking  beyond  the 
confines  of  the  instruction 
setting  (Carlton  College, 
2011). 

Introduce  students  to  a  problem 
associated  with  the  training  content  and 
have  them  openly  discuss  relevant 
associated  issues.  Choose  two  or  more 
individuals  from  the  class  to  act  out  a 
hypothetical,  real-world  scenario.  Give 
each  of  the  participants  associated 
information  for  their  part  in  the  role 
playing  and  have  the  scene  acted  out  in 
front  of  the  remaining  students.  Have 
an  open  facilitated  discussion  of  the 
role  players’  actions  having  students 
critique  their  performance  and  offer 
suggestions  and  justifications  for  how 
they  would  have  approached  them 
differently. 

B-3 


HOTS 

Instructional 

Method 

Description 

Purpose 

Example 

Analyzing, 

evaluating, 

deciding, 

planning, 

predicting 

Simulation 

Mimicking  or  imitating 
an  actual,  possible,  or 
probable  real-life 
condition,  event  or 
situation  often 
involving  equipment 
such  as  computers, 
electronic  simulators, 
or  actual  equipment. 

To  have  the  student  develop 
cognitive  skills  such  as 
reasoning  and  decision¬ 
making  while  learning  the 
associated  technical  and 
behavioral  skills  for  the 
content  being  trained. 

Have  student(s)  conduct  an  exercise  on 
actual  or  simulator  equipment  while  the 
trainer  and  other  students  observe  the 
actions.  Upon  completion  of  the 
exercise  have  the  participants  conduct  a 
review  of  their  own  actions  followed 
by  peer  critiques  and  suggestions  for 
alternative  actions  and  improvements. 

Analyzing, 

evaluating, 

deciding 

Problem 

solving 

Problem-solving  is 
most  often  case-based 
learning  where  students 
seek  to  find  solutions  to 
problems.  It  involves 
taking  what  they  know 
and  applying  it  to 
discover  what  they 
don't  know  (Major, 

1998).  May  include 
methods  such  as  role 
playing,  simulations, 
debates,  analysis  and 
reflection,  and  group 
projects  (Blackmon, 
Hong,  &  Choi,  2007). 

Students  are  involved  in 
active  learning  through 
discussions  in  order  to  solve 
problems  or  resolve 
conflicts.  Provides 
opportunities  for  analysis, 
proposal  of  solutions, 
evaluations,  and  decision¬ 
making  (Blackmon,  Hong, 

&  Choi,  2007). 

Present  students  with  a  training  content 
related  problem  with  ambiguous  or 
incomplete  data  in  order  to  force 
critical  thinking  and  analysis.  Divide 
students  into  small  groups  to  search  for 
a  solution  providing  additional 
information  during  the  progression  of 
the  exercise  or  as  requested  by  group 
members.  Have  each  group  explain  its 
solution  to  other  groups  for  critical 
critique. 
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HOTS 

Instructional 

Method 

Description 

Purpose 

Example 

Observing, 

examining, 

criticizing, 

analyzing, 

determining 

Case  study 

In-depth  investigations 
of  a  single  person, 
group,  event  or 
community  (McLeod, 
2008).  Presented 
through  narratives, 
situations,  data 
samplings,  or 
statements  that  present 
unresolved  and 
provocative  issues, 
situations,  or  questions 
(Indiana  University, 
2012). 

To  allow  students  to 
investigate  a  topic  in  detail. 
Often  sheds  light  on  aspects 
of  human  thinking  and 
behavior  and  can  help 
generate  new  ideas  and 
theories.  To  challenge 
students  to  analyze,  critique, 
make  judgments,  speculate, 
and  express  opinions 
(Indiana  University,  2012). 

Present  students  with  a  historical 
example  of  a  training  content-related 
event.  Provide  details  of  the  situation 
to  include  scene  setup  before  the  event, 
additional  data  and  complicating 
factors  associated  with  the  event, 
actions  taken  before,  during  and  after 
the  event,  and  ultimately  the  outcome. 
Use  questioning  and  discussion 
techniques  to  stimulate  critical  thinking 
and  promote  student  discussion.  Have 
students  analyze  and  critique  the 
actions  of  those  involved  in  the  event. 

Evaluating 

and 

organizing 

Concept 

Mapping 

A  graphical  technique 
to  organize  and 
represent  information 
(Novak  &  Canas, 

2006). 

To  engage  student  in 
actively  making  sense  of 
and  organizing  relationships 
between  concepts,  ideas, 
etc. 

Provide  students  with  a  list  of  key 
concepts  and  have  them  diagram  the 
concepts  in  a  concept  map  illustrating 
how  they  are  related;  following  this 
have  them  compare  their  maps  to  those 
of  their  peers  and  explain  why  they 
organized  their  map  the  way  they  did. 
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HOTS 

Instructional 

Method 

Description 

Purpose 

Example 

Choosing, 

evaluating, 

arguing, 

criticizing 

Debates 

A  structured  contest  of 
argumentation  in  which 
two  opposing 
individuals  or  teams 
defend  and  attack  a 
given  proposition  by 
presenting  and 
contesting  varying 
points  of  view  with 
regard  to  an  issue 
(Cruickshank,  Jenkins, 

&  Metcalf,  2011), 
(McCambridge,  n.d.). 

To  have  students  consider 
not  only  the  facts  of  a 
situation  but  the 
implications  as  well.  To 
have  students  think 
critically  and  strategically 
about  both  their  own  and 
their  opponent's  position 
(McCambridge,  n.d.). 

Divide  the  students  into  two  groups. 
Choose  a  debatable  issue  associated 
with  the  training  content  and  have  one 
group  jointly  research  the  pros  and 
cons  of  one  side  of  the  issue  and  the 
other  group  research  the  pros  and  cons 
of  the  other  side.  Choose  three 
students  from  each  side  to  conduct  the 
debate.  Position  the  two  teams  at  two 
separate  tables  in  front  of  the  remaining 
students.  Have  the  remaining  students 
develop  a  list  of  thought  provoking 
questions  to  be  presented  for  debate. 
Provide  the  questions  one  at  a  time  and 
give  each  team  a  chance  to  respond. 

Planning, 

justifying, 

deciding 

Brainstorming 

Students  spontaneously 
offer  potential  solutions 
to  a  problem, 
regardless  of 
feasibility,  all  input  is 
recorded  with  initially 
no  evaluation  of  ideas, 
ideas  are  evaluated  in  a 
subsequent  analysis 
(Bacal,  2015). 

Uses  students’  diverse 
experiences  and  encourage 
all  learners:  to  get  involved 
and  think  both  creatively 
and  critically,  to  think 
beyond  immediate  day-to- 
day  issues,  to  generate  ideas 
from  the  group  and  learn 
from  each  other  (Bacal, 

2015). 

Define  a  training  content-related  topic 
for  contemplation  such  as  seeking  to 
find  a  solution  to  a  problem,  answering 
a  question,  or  seeking  to  invent 
something  new.  Present  the  idea  and 
encourage  discussion.  Record  all 
responses  without  prejudice,  encourage 
participation,  and  allow  for  no 
criticism.  After  a  set  time  limit  display 
all  ideas,  eliminate  bad  ideas  and 
critically  discuss  the  remaining  ones. 
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HOTS 

Instructional 

Method 

Description 

Purpose 

Example 

Comparing 

and 

contrasting, 

deciding 

Discussion 

Discussions  occur 
when  a  group 
assembles  to 
communicate  with  one 
another  through 
speaking  and  listening 
about  a  topic  or  event 
of  mutual  interest 
(Cruickshank,  Jenkins, 

&  Metcalf,  2011). 

To  stimulate  critical 
thinking  and  challenge 
students  to  think  deeply  to 
express  their  ideas  clearly. 
Students  are  active  in 
learning  with  opportunities 
to  formulate  principles  in 
their  own  words  and 
suggest  applications  of 
these  principles 
(University  of  California, 
Los  Angeles,  2011). 

Describe  a  situation  where  an  item 
must  be  selected  from  a  choice  of  two 
items  (such  as  a  potential  replacement 
sidearm  for  Army  Soldiers).  Through  a 
facilitated  discussion,  have  the  students 
assemble  a  list  of  characteristics  for 
comparison  and  subsequently  identify 
the  pros  and  cons  of  each  item  as 
compared  with  the  other.  Discuss  the 
justifications  for  choosing  the 
characteristics  for  comparison. 
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